This week on Swarm Stack, I sat down with roboticist Dr. Merihan Alhafnawi to talk all things swarm art, architecture, and human-swarm interaction. In this conversation, we explore her latest Science Robotics paper, Swarm Garden, and delve into how robot swarms can be used for creative expression.
About Merihan Alhafnawi
Dr. Merihan Alhafnawi is a Postdoctoral Researcher at Princeton University, working with Professor Radhika Nagpal at the SSR lab to implement embedded swarms in self-adaptable structures. Her research focuses on the hardware design and build, as well as algorithm development, of social robotic swarms. Ranging from creating robotic elements on façades to make them responsive, to creating smart robotic sticky notes for brainstorming, her robots have been used by more than 500 people. Merihan obtained her PhD in 2022, from the Hauert Lab at Bristol Robotics Laboratory, supervised by Professor Sabine Hauert and Dr Paul O’Dowd.
Transcript (with a few edits)
Sabine Hauert: Hi, Merihan. Welcome to Swarm Stack.
Merihan Alhafnawi: Hi, Sabine. Thank you so much for having me.
Sabine Hauert: It’s so good to have you on the show. We’ve obviously worked together in the past, and you’ve done beautiful work over the years in swarm creativity and using swarm robots as a new way to interact with people, which I think is really exciting. Congratulations on your new Science Robotics paper.
Merihan Alhafnawi: Thank you.
Sabine Hauert: Do you want to tell us a little bit about Swarm Garden?
Merihan Alhafnawi: Swarm Garden is this collaboration between swarm roboticists, civil engineers and architects. We were trying to figure out how to combine these fields in order to make adaptive and responsive architectural façades. Currently, most of our buildings are static. Can we add robotic elements to make them responsive not only to us, the people inside the building, but also outside, to the environmental changes and climate changes?
Swarm Garden came out of this collaboration, and it’s this architectural swarm that has elements that you can, for example, place on windows, and each element opens and closes. Then these elements that we call the SG bots, Swarm Garden bots, each open and close, and they coordinate with one another to figure out the best lighting for the room based on the user’s preferences, the lighting conditions, and the sunlight intensity. And because these systems are going to exist in our spaces, we also looked at how they enhance well-being of occupants - how they can bring a sense of joy.
We also explored Swarm Garden as a form of creative expression. We exhibited the bots at Princeton, and we had people interact with them. We created this wearable people could use to influence the swarm, based on their hand gestures. We gave that wearable to a dancer who performed live in the exhibition, and twice later. When we interviewed her, and she said that the Swarm Garden enhanced her creativity and made her think differently about what dance is and how technology can also enhance dance and creativity.
Sabine Hauert: It sounds magical. It’s a lot of work to make the actual robots because you design everything from scratch. Do you want to describe the robot modules themselves? I’ve seen them in action and they’re really beautiful - they look like little flowers. Could you also describe the modularity of how you can add them and assemble them in different ways – help us imagine them?
Merihan Alhafnawi: Sure. Each robot is made of this very thin plastic sheet that when we confine it into a circular opening, or pull it through a circular opening, the sheets kind of crumble. And then they fold. When the sheet is flat, it blocks the light, and then when we pull it, it creates this flower shape and then allows light to enter the room.
The postdoc at the time who was looking at this with me, who’s now a professor at Northwestern - Professor Lucia Stein-Montalvo - had previously studied the properties of this thin sheet when it folds. And we thought that we could turn it into a robot and make many modules. It becomes modular because each module is independent, so you can create different shapes with it and make them into different reconfigurations. It’s highly reconfigurable. And each module is simple - it has one motor. It has a couple of sensors. They can be put together really quickly and scaled really fast.
Sabine Hauert: When you came to do this project in Radhika Nagpal’s lab, having worked on swarms and creativity and interactions with humans, was that the starting point of Swarm Garden? And then you realised, wow, this is actually a wonderful interface for architecture. Or did you start with architecture and then it happened that it has all this creativity to it as well? How do you go about designing these tools that can be used in so many ways?
Merihan Alhafnawi: That’s a great question. I think it actually came first from architecture. Radhika Nagpal, who I work with, she had this collaboration with Professor Sigrid Adriaenssen from Civil Engineering, and they were thinking about how to merge the research in order to create adaptive façades. And then Radhika said that she needs human-scale robotics - not micro robots or small robots, but actually robots that scale in big sizes. And then when I came into the project, I thought, these robots are going to exist in our spaces, so they need to have a human interaction aspect to them. This is where we started thinking of not just purely the hardware design, of how we can make robots that adapt to sunlight, but also how can we make them adaptive to people, and how can we make the interaction interesting? That’s also when we started thinking about the creative expression applications. If these are going to exist in our spaces, we need to make them beautiful, and we need to make sure that people would be comfortable and happy while interacting with them.
Sabine Hauert: Were they? Did they enjoy it?
Merihan Alhafnawi: They were. In the exhibition, we surveyed people and asked them to describe the robot or their experience in general, just with one word. And we got about 96% positive sentiments from the responses. People were really happy. They used a lot of words like cool, amazing, floral. And I think generally everybody was very animated and happy in the exhibition, especially when the dancer danced with the robots. That was also an impressive piece of the exhibition.
Sabine Hauert: I was just thinking about the floral element. You always have flowers on, and you do today! And then, you know, you often say that of your pets as well. I’m sure Mango’s somewhere around.
Merihan Alhafnawi: Even my dog has a lot of floral things.
Sabine Hauert: That’s amazing. I also wondered where next with this, because now you’ve done the proof of concept that this can integrate into human environments. Is this something that you see could translate to architecture, or is there just so much more research that needs to be done in these smart spaces?
Merihan Alhafnawi: It’s true. As part of the Science Robotics paper, we did some experiments where we put them on a real office window. We stacked around 4 rows of robots, so 16 in total. We saw how they interacted to sunlight and, using an opinion dynamics model, how they took into consideration not just my preference as a user and as an occupant of the room, but also how much sunlight there is out there. And they reacted successfully to sunlight. But then the next step is actually to work with architects to see how this can scale to big buildings. How can it be integrated into buildings? And also, algorithmic-wise, the next step is to see how they adapt over time to different user preferences. So how can we have customisable experiences for every user that is in the room, or different rooms, and how it learns from people’s behaviour?
Sabine Hauert: There are beautiful concept drawings from the architectural side in the paper as well. So you can really start to imagine what this might look like at larger scales. It’s very cool. This is not your first robot that interacts with humans or swarm that interacts with humans. In the past, you’ve worked with other systems like the Kilobots, using them as canvases that people could paint on in an interactive way. And you’ve designed 100 robots in lockdown called the Tiles that can assemble into MOSAIX. What have you learned through designing all these systems? And what do you hope for the future in terms of where this field is going?
Merihan Alhafnawi: So the robotic canvas work that I did with the Kilobots and the tiles, which are part of the system called MOSAIX – this was all work I did when I was your PhD student.
Sabine Hauert: We used them this weekend in my neighbourhood actually, to ask the neighborhood what they thought about robots for the future. And all your little robots were post-it notes for my neighbours.
Merihan Alhafnawi: Oh my God, you need to send me pictures.
Sabine Hauert: I’ll talk about that another time. Let’s carry on. So, what have you learned and where is it going?
Merihan Alhafnawi: When I first worked with the Kilobots, it was super early stage research that I did as part of my master’s with you. And I learned a lot about how the robots themselves can interact with one another, and how I can exploit these interactions between the robots and what they can sense from in their environment, so that I as a user can interact with them. As we know, swarm robots, they’re decentralised. So, when we want to add a person, we don’t want to break this decentralisation. Robotic Canvas taught me a lot about how to exploit the environmental and the local interactions to add a human aspect to the system. So that was great.
We did these paintings where people could collaborate with each other to create paintings with the Kilobots. They could also move and be used as a sculpting material to create shapes. While the robots were actually not made for interactive purposes, throughout this project, we transformed them into interactive robots. They were very limited in what they can do, and they needed specific conditions, for example, ambient light.
So that’s how the MOSAIXs came into life. How can we design robots that are highly interactive, that we can deploy in very different environments and that people could interact with really intuitively? That’s how we had the idea to do it with a touch screen, because everybody knows how to use a touch screen. It’s not going to take them a long time to learn, but also we could walk people through the interaction, through the touch screen. We could show different kinds of videos or pictures or buttons.
Then that’s when we took it out to shopping centers, to a museum, to schools. And we did learn that having a highly interactive system makes deployment really easy and makes interaction really easy. And at some point, even museum staff were able to deploy the robots on their own. I would take breaks and I would go, for example, have lunch and they were able to still keep using the robots and explaining to people how it works without a lot of training from me. One of the insights is that highly interactive robots, highly intuitive robots, are very important, not just for the users, but for the people deploying them. And, we learned a lot about how multiple people can interact with multiple robots. In previous research, there were a lot of human swarm interactions, which is one person interacting with the swarm.
Sabine Hauert: Like an operator.
Merihan Alhafnawi: Exactly. With MOSAIX, the beautiful thing about it is that it’s opened up the opportunity for us to explore multiple people interacting with multiple robots in public settings, with people who might not be technical. And we gained a lot of great insights about what people thought worked in the design, what they didn’t think worked, and how we can enhance our systems and how we could add more interactive aspects to our system.
Sabine Hauert: Because you keep designing new hardware, do you think we need to rethink hardware for this type of interaction? Is there something that we should be thinking differently about in terms of what swarms look like when they’re in these interactive spaces?
Merihan Alhafnawi: Something that I really appreciate in your research as well, and I think is very important, is how we have focus studies that tell us what people really want. Because even though I worked with three different swarm systems, two of which I designed, they look completely different. So, the Kilobots are these small robots that don’t have wheels and they buzz and vibrate to go to places. The Tiles are like those touch screen robots that have wheels and can integrate into our environment. And then the Swarm Garden is these robots that you can put on your windows or in your space. They don’t have wheels. They’re like robots in a grid. Completely different designs because they’re for completely different purposes.
It’s one of the challenges of, interactive swarms - thinking about the form that the robot takes. It’s not your usual humanoid, it’s not your usual drone. Now they have to be very creative and very suitable for the task in hand. I think keeping an open mind about what a robot is, what makes a robot, and that we could have really out-of-the-box designs that really work with people. And I think people in this context can really help us - tell us what works for them and what doesn’t. And that’s why - going back to the first point - focus studies are really important. Having early on deployments where people can give us their feedback is also very crucial.
Sabine Hauert: That makes me dream, and hopefully makes people dream more broadly about what swarms could be like within their everyday environment. Thank you so much, Merihan, for joining us on Swarm Stack.
Merihan Alhafnawi: No, thank you. It’s my pleasure to be here.
Special thanks to Ella Scallan for editing this post!
