Building Context with Probabilistic Models
Understanding the semantics of the environment has potential applications for enabling robots to customize for their users and help them in everyday life. Popular machine learning tools such as Large Language Models (LLMs) have proven to be extremely useful for extracting and understanding semantic information from language. Yet they suffer from problems such as hallucinations, jailbreaking, and the physical sizes of the models making them unable to be run on the edge. This research compares methods that are not reliant on predefined information of the environment to replace LLMs for learning contextual information to make them more predictable, safer, and accurate. I adapt probabilistic methods, including moving average, bayesian inference, and expectation maximization, to ground objects based on the semantic information of the environment. These all exist under one generalizable package to be able to be deployed on any robot under any environment. These methods are then evaluated based on how well they do against an LLM for a variety of tasks. Finally, I built the WorldGenerator and WorldAction software architecture around the PyRoboSim simulator.
Advisor: Professor Kevin Leahy (WPI)
Committee: Professor Fiona Yuan (WPI) and Professor Carlo Pinciroli (WPI)