Connectionism: Some Basics

• Where the connections come from
  • The modeler builds them in
  • They evolve
• Where the weights come from
  • The modeler builds them in
  • They are learned; initial random weights
  • They evolve
• Net input to a unit, usually
  , X the external input
• Unit activation and output
  • Corresponds to firing rate of neurons or to pattern of firing across group of neurons
  • Clamping units: holding their activation constant (because of external input) regardless of input from other units
  • Continuous vs. discrete (usually binary)
  • vs. [-1,+1] vs. [0,1]
  • Activation and output functions
    • Identity
    • Unit threshold (bias): constant value subtracted from (added to) input in computing activation
    • Threshold (step) function
    • Sigmoid (squashing function), for the [0,1] case:
      , T a temperature parameter
      
    • Stochastic functions: treating the sigmoid as the probability of turning on (or flipping) the unit
  • Activation updating
    • Asynchronous: sequential or random
    • Synchronous: update units in parallel

[L645 Syllabus]