EEG-to-Image Generation for Brain Injury Rehabilitation (ArXiv)

OVERVIEW

During my EEG computer-vision research internship at Harvard Medical School, I helped refine DreamDiffusion, a model designed to convert EEG signals into images. Our team resolved major environment and dependency issues that made the original GitHub version nearly impossible to run, migrating the entire pipeline into Google Colab and debugging model-architecture and computer-vision components to ensure reproducibility. This process taught me how to systematically diagnose errors within complex deep-learning systems. We also evaluated a variety of classical and deep learning approaches, including SVMs, feedforward DNNs, CNN-based encoders, and generative models such as GANs and VAEs, for EEG-to-image reconstruction. While each model captured limited aspects of the signal, they struggled to robustly map EEG data to meaningful visual representations due to EEG’s low signal-to-noise ratio, nonlinearity, and complex temporal dynamics. Understanding these limitations, and why DreamDiffusion instead leverages masked-signal pretraining and alignment within CLIP’s latent space, deeply shaped my understanding of how to design models that translate noisy biological signals into meaningful visual outputs.

WHAT I DID

• Refactored and debugged the DreamDiffusion codebase into a reproducible Google Colab pipeline, resolving dependencies, setup errors, and environment incompatibilities.
• Implemented EEG preprocessing and representation pipelines, dealing with EEG's low signal-to-noise ratio and temporal structure
• Evaluated numerous ML and Deep Learning paradigms, and explored CLIP's multimodal latent space, enabling Stable Diffusion to generate images via EEG-derived data.
• Worked with a team of 3 other peers to write a 13-page research paper, exploring our findings and documenting the methodologies and insights we made.

RESULTS / IMPACT

• Lowered the barrier to entry for EEG generative modeling via providing clean documentation, executable notebooks, and preloaded data, enabling easier experimentation for future studies.
• Demonstrated diffusion-based latent-space generation is more effective for EEG-to-image generation than CNNs, GANs, VAEs, or SVM models.
• Wrote a paper published to ArXiv as a pre-print.

LESSONS + NEXT STEPS

• Utilize University of Pennsylvania Venture Labs and other entrepreneurial sources to make this a hospital-utilized product.
• Integrate wavelet-based and time-frequency features to better capture transient EEG dynamics