Now, a puff from reference 4 has alerted me some rather different work on reconstructing faces from brain waves, one of the many lines of attack of the many scientists trying to decipher the waves generated by the brain. It seems that quite a lot of facial reconstruction can now be done from both fMRI scans (reference 2) and EEG (reference 1), it seeming that the superior temporal resolution of the latter compensates for its inferior spatial resolution. Nevertheless, it is something of a surprise to many that an aggregating, broad brush tool such as EEG contains enough echoes from the detail of face processing to reconstruct those faces.
I think it is fair to say that these scientists are as interested in where and when the work is done in the brain as in these reconstructions – while I, for the moment anyway, am only interested in the latter.
The diagram offered above takes various liberties, but I hope it captures the general idea, with an important intermediate construct being the face-space, a multi-dimensional real vector space into which one maps faces. Each dimension – and there might be of the order of a hundred of them – corresponds to a feature and the idea seems to be that from any point in the vector space, corresponding to a face, you reconstruct the image of that face (or, more precisely the difference between the image of that face and an average face) by forming the weighted sum of the corresponding features, with the weights being the coordinates of the point in question. And where a feature is not something like a nose or an ear, rather something much more abstract, in the words of reference 3, a ‘global whole-face image structure’. And where faces have been standardised a bit, in the words of the supplementary information for reference 2: ‘these images were spatially normalized with the position of the eyes, were cropped to eliminate background/hair, and were normalized with the same mean and root mean-square contrast separately for each colour channel in CIEL*a*b* colour space’. For which last see reference 7.
Again, it is something of a surprise that one can do this, that one can build a vector space in which one can simply add up bits of abstract face and get a real face. Perhaps less of a surprise when one thinks that this is something like what one does with the two dimensional Fourier transform of any image.
All of which leads onto the process which follows.
Step 1, make a collection of images of faces. Perhaps process them a bit to bring them into line with some standard. The choice of images and faces will depend on what exactly it is that one it up to. Who is paying for the work to be done – in which connection the supplementary information for reference 2 included the titbit: ‘Portions of the research in this paper use the FERET database of facial images collected under the FERET program, sponsored by the Department of Defense Counterdrug Technology Development Program Office’. Presumably the people dreaming up the acronym liked the association to the traditional job of ferrets. See reference 6.
Step 2, make a real face-space for those images and derive the corresponding features. A proceeding which requires both statistical and computing power. And a lot of which involves principal component analysis (or one of its relatives) to reduce the dimensionality of the raw images.
Step 3, have a group of people look at the faces while sitting inside an fMRI scanner or while wired up to an EEG recorder. Some care is needed in devising the right experimental protocols here.
Step 4, process the material coming out of the scans to make a virtual face-space. A proceeding is much less direct than step 2 and which requires even more statistical and computing power.
Step 5, conflate the two face-spaces. Or put another way map one onto the other – with the papers referenced talking, in this connection, of the Procrustes the Greek. This enables one to compute the virtual features from the real features. This being the bit which gets one from brain waves to pixels.
Step 6, scan the brain waves arising from some face, not necessarily one that has been used before, compute its position in the virtual face-space and reconstruct the image. With, it seems, some success.
So while one is not exactly conjuring the image out of the scanner, one is not conjuring up the subjective experience, one is using a few props to translate the brain waves into pixels, rather as your Latin teacher might have used his accumulated knowledge to translate Virgil into English. One is heading in right direction. Which, assuming that I have got it right, I find very impressive; no wonder that it was flagged up at reference 4.
One is also learning something about how the brain does all this. In particular that what it is doing amounts, in some sense at least, to using a face-space, rather as one might in a computer. And something about how its face space relates to the one that the computer makes do with.
PS: pursuing the analogy, I observe that translating the Virgil would be a great deal harder if that one Latin text was all you had to go on. No Rosetta stones. Furthermore, that your subjective experience of the English translation, however good, is going to be very different from that of a Latin speaking contemporary of Virgil.
References
Reference 1: The Neural Dynamics of Facial Identity Processing: insights from EEG-Based Pattern Analysis and Image Reconstruction - Dan Nemrodov, Matthias Niemeier, Ashutosh Patel, Adrian Nestor – 2018.
Reference 2: Feature-based face representations and image reconstruction from behavioral and neural data - Nestor, A., Plaut, D. C., Behrmann, M. – 2016.
Reference 3: Face-space architectures: evidence for the use of independent color-based features - Nestor, A., Plaut, D. C., Behrmann, M. – 2013.
Reference 4: http://www.kurzweilai.net/.
Reference 5: http://psmv3.blogspot.co.uk/2017/11/reading-brain.html.
Reference 6: https://www.nist.gov/programs-projects/face-recognition-technology-feret.
Reference 7: http://dba.med.sc.edu/price/irf/Adobe_tg/models/cielab.html.
No comments:
Post a Comment