Part 1: why CV and Part2 : how comparse to brain?
ML in animals:
- Detection of objects - useful for fish movement/herding/migration
- Detection of behavior - classifying behavior of ants, time spent walking/cleaning etc
- Further on this, using detections of behavior alongside geno/phenotypes to see the ‘mutant/deviant’ types/expressions
- Does a fly keep swerving over and over instead of flying straight/perching?
- Further on this, using detections of behavior alongside geno/phenotypes to see the ‘mutant/deviant’ types/expressions
- Could be used for knockouts
package: “sleep” - detects key points
- Useful for posture of animals
- Tremors in mouse hands (drugs/genotype tests)
- Postural displays in mating
State offield:
- Lots of tools
- Limitation not in tech but in training and ease ofuse
- People designing computer vision approaches need to incorporate them into user-friendyl interfaces, people studyign animal behavior need to understand how computer vision works
Classical Approaches: Explainable
Deep learning: Losesthat explainability
Classical CV:
- Have animage ofa ocean, and remove that from current picture. left is the boat on ocean in current image
- Super explaianbale, efficient
- Subtract base matrix from current matrix:
- Cons:
- Needs high contrast bg
- Cant find differences that arent that different in color (IE white shirt on white background)
Motion:
-
Pre or post processing
-
Optical flow
- Similar to the above classical CV, keep a running track of background ‘moving average’ of motion
-
Tracking algorithms
- Can also do more ‘big ML’ and detect objects on each frame, and worry about running tracking algorithms between frames
In brain:
- usually fixation
- foveating+smooth pursuit
- however,still able to fixate on onething and track motion elsewhere
Part 3: CVin his research
‘mechanismsofsocial behavior’
phd work with birds,fish fighting,clone wars?
birds
where does variation in courtship come from?
- Does the female bird change?
- Does the song chage?
CV allows for mass recording of courting stats
- Allows them to demonstrate that postural display occurs beforethe whistle iseven heard
- the clickbeforewhistle more important?
fish
Amaon molly
behaveAI
running background preprocessing, and then labeling different frames in differentcolors
has theadvantage of being used to make labeling easier, and cleaning data for future ML / data cleaning
- Labeling hard for fish, small hard to see
- behaveAI background filter is great in
“behavioral ecology”
fish fight
winner effect
- proven that giving a ‘free win’ makes the next matcha more likely win
- (cripple oppenent fish on first round then have actual measruement in 2ndround)
idTracker
losing track of identities (crossing/mix ups) leads to error propogated thru whole vid
idTracker trains on fish identities, so we can be certain to not mix them up.
aperkes[at]ucd
R-CNN?
understand the biases
difficult to understand “WHY” chatgpt and CV iswrong. it is CRITITICAL to know where/wy is wrong to verify