Kathrine

Thank you for joining me today. To start, would you mind introducing yourself, your field, and the main question your research group studies?

Professor Wu

Yeah, sure. So I'm Jason Wu. I'm an assistant professor at Purdue in the Department of Computer Science, and I'm the director of the CIDR Lab.

So CIDR is one of these acronyms, right? So CIDR stands for Computational Interaction Design Engineering and Research, and it sort of represents some of the research questions that we're interested in. So we try and use computational approaches like optimization, like data-driven modeling, right?

And we try and solve problems, human-centered problems in HCI. So these are things like how should we interact with computers? How do those interfaces get created?

How do we design things, right? So we're kind of at this intersection of AI and HCI. And I would say, like, there are kind of two categories of ways that this could happen.

Like one is how do we use AI to make this HCI, like, better, right? So we can actually see a lot of examples of this all around computing. So like recommendation systems, gestures, all these things.

And especially, like, a lot of the work that we do is for accessibility purposes, like, how do we get AI to describe things? And then on the other side, what we're doing a lot of the time is, like, how do we use HCI to improve AI? So, like, how do you know if an AI is useful?

How do you know if it's good? It probably needs to do things that humans want it to do. And there's a lot that we're trying to do in, like, how can we get AI to learn from human experts, like designers?

How do we like measure AI, like how good it is in a really principled and reliable way? So we kind of have projects in both of those areas, but that's kind of a flavor of the things that we're doing.

Kathrine

So you teach a course called computational interaction. What is that? And how does it use human behavior design and machine learning to create better interfaces?

Professor Wu

Yeah, so computational interaction, I would say, is one way of looking at it is, like, it's an area of HCI. I think I personally look at it as almost like an approach for solving or, like, trying to, yeah, work towards human-centered problems. So, like, the goal of a lot of HCI and human-centered design is to build interfaces that that would be good for a certain user or group of users.

So an example of this would be, like, a website, obviously, or, like, a keyboard. And there are, like, many actually hidden variables that go into how a keyboard is designed or, like, how a website is designed. Like a keyboard, especially, right, you might think it's, like, everyone is using the same keyboard.

But if you think about, like, on your phone, like, how big should the keys be, right? Like, how far should they be spaced? Sometimes when you click, like, some letters actually have, like, a little bit of a bigger tolerance of, like, what will register on that letter.

And, of course, for, like, websites, there's, like, a ton of different layouts you can do. So traditionally, a lot of this has been done manually, like, in the human-centered design process. You'd have, like, an expert design these things and test it out with users, use their feedback to sort of, like, guide your revision, and you can kind of repeat this process.

So computational interaction sort of thinks about ways, like, how can we embed computation into this process and even make it, like, make parts of this, like, much easier? So I can give you one example, right? So instead of first designing a keyboard or website and then getting people to use it and then, like, collecting data or feedback from them to, like, revise the keyboard, what if we first collected a data set of just what people thought looked good in general, right?

Like, if you had an image of something, image of a design, what makes an image look good? What makes a keyboard usable? Like, can we predict, can we simulate people typing different words based on the, you know, distance between keys?

And what if we first started with that model and then ran, like, an optimization algorithm can actually search through different possibilities of, you know, these parameters and, like, how a website would, might look like, these candidates, right? So that is one example, and it turns out that for some applications, like in machine learning, it makes a lot of sense, because machine learning needs to be trained with, like, you know, really fast computers that are evaluating this objective really fast, and we don't really have time for experts to rate, like, every single possible output in the model. So, yeah, it makes a lot of sense in a lot of ways.

It also has the weaknesses for sure, but, like, the way I see it is it's, like, an approach to HCI that sort of extends these goals to sort of, like, new domains, and that's a little bit of what we cover in a class and also what we do in our research.

Kathrine

So your early research involved a lot of smartwatches, wearable computers, and other small devices. So what did those projects teach you about accessibility and user interfaces?

Professor Wu

Yeah, so I think from, so to myself, they actually follow a pretty consistent and kind of, yeah, consistent trajectory or shared thread. It kind of seems like different things, but they actually share a lot of the same challenges. So I can kind of give you an example where I used to work on smartwatches and, like, head-worn displays, like these wearables, like you said, and one of the biggest problems for those is, like, how do you efficiently, like, input text into that, or how do you, like, efficiently select buttons on that?

And it's really hard because it's, like, a really small screen, and there's no, like, full-size keyboard, right? So, like, and you might be using, you might have gotten, like, a notification while you're doing something else. So there's all these problems that in wearable computing that are around this problem of, like, how do you efficiently talk to your computer, respond to notifications, things like that.

And that's actually a lot of the same, there's, like, a huge overlap with accessibility, right? So if you had motor tremors, or, like, let's say you could only use one hand, you had someone who could only use one hand to do something, that, like, the sort of solutions to those, like, imagine if you had, like, a better gesture-based way of communicating with your computer, or even voice-based input, those technical solutions are actually very much in line with each other many of the time. So, and so, in fact, a lot of these, we have this term called situational impairments.

So, like, instead of, like, you would have this impairment that only exists in a situation. So if you're riding a bike and you need to respond to a notification, or you're talking with someone or your computer in a very loud environment, right, like, you would face the same sort of challenges as, you know, someone who's deaf or hard of hearing, at least in that instant, or something similar. So that is, like, a lot of the shared, there's, like, a lot of shared technical foundation between them.

And, in fact, right now, I do a lot of work in sort of design, like, how do we get an LM or a machine to generate a good design? There's actually a lot of overlap there, too, because, like, if you think of the question, right, is this interface or is this object accessible? That's, like, almost not even a yes, no question.

Like, accessible for who, right? Like, that might mean different things to different people. And is it something that they could just barely use?

Like, you know, in theory, they could use it, but it, like, it was super hard for them to use. Or was it, like, something that was really a breeze for them to use, right? So this is also shared with design, where, you know, you might want to be able to quantify this and make decisions.

So, yeah, I guess what I'm trying to say is accessibility, I think I have a pretty broad view of it. It's, like, how do you make computers accessible to all people across all contexts? And so that's what underpins a lot of my research, yeah, over the years.

Kathrine

Next, if an AI looks at a screenshot of an app or website, what would it need to understand beside where, like, the buttons, menus, and pages are?

Professor Wu

Yeah, so, like, one thing I always like to think about is at the end of AI somewhere, right, there's a human, right? Like, at some point, some chain reaction may be called AI, and we have to sort of trace back all the way to where the human was. And I think, so that maybe depends on what the AI needs to be able to discern from a screen.

A lot of the work I did during my PhD was for accessibility purposes for people who are blind or have low vision. And in that case, you would want, like, you would want a model to look at a and efficiently describe what the screen can do and also surface its functionality in a way that could be, you know, easily accessed by that blind person. So that is, like, one example of a use case.

Another example that I'm looking at right now is more for designers, right? Like, maybe you want to have a designer that maybe they use an agent to generate, like, a website. Or maybe someone else, like a collaborator, gave them a website.

What do they think? Like, what makes them think it's either good or not? And the model that, the AI that, like, in that scenario probably needs to pay attention to different things, like the colors, the patterns, the styles, all of those different things.

So I think it really depends. But the good news is that we have bigger and more powerful models that are able to kind of do many of these same things under the same architecture infrastructure. And that's a really exciting time.

Kathrine

So in some of your research, in some of your recent projects, designers used comments, sketches, and direct edits to improve AI-generated interfaces. So what does the AI itself learn from that kind of feedback?

Professor Wu

Yeah, so I think you might be talking about a paper I wrote called Improving User Interface Generation Models from Designer Feedback. So that's a bit of a long title. And what we called it, like our kind of pet name for that project was Reinforcement Learning from Designer Feedback.

So if you ever heard of the term RLHF, or Reinforcement Learning from Human Feedback, the purpose is, how do we get models to be, quote unquote, aligned? So this basically means, like, can a model understand how to respond to responses in a way that we, as the makers of the model, consider good as well? So this can mean a lot of different things.

Like, it can mean something that's, like, safe, right? Like, it shouldn't tell you how to build a bomb, or, like, it shouldn't lie. In the design context, it sort of means, like, how do you make something that we would consider, you know, something that looks good or, you know, doesn't look good or is usable?

And RLHF today is done in, I think, a pretty, like, in a way that's a little bit suboptimal for design. So if you ever used GPT, chat GPT, sometimes you'll see, like, this, like, a small thumbs up button at the bottom, thumbs down button, or sometimes it'll, like, show you two, like, versions of the responses, and you can click, like, oh, I like, you know, A or B better. And it turns out, when we had tried to apply these same, like, generic strategies to design, it was really bad at capturing what designers thought was actually a symbol of good design or aspects of good design.

Because, basically, you don't get a lot of information from these, like, really low fidelity responses. And the other problem is that, like, people will disagree with each other a lot for different reasons. So one person might think B is better than A or A is better than B, and they both have valid reasons for that.

They're, like, have different tradeoffs that they have evaluated. And you would end up with a lot of noise, a lot of disagreement, which makes it really hard to train a model. So, and the other problem is, like, if you ask designers what they want to do in the next five years for their job, they don't want to be, like, rating things with thumbs up or thumbs down all the time.

So our paper was trying to basically learn what makes a good design from the things that designers are already doing. So you can get a lot of signal from interactions such as revising an interface. So let's say the model generated something, and there was some problem with it.

So, like, you can kind of go into your editor and actually change the font, make things aligned. And now you have this sort of, like, before and after that the model can learn from. So there's all these different ways that you get really good high quality signal, and the designer is actually, like, happy to do this as a part of their job.

They've already been doing it, and they're very used to it, and the model learns a lot better. So that was, like, really the point of the project, and I think it worked out pretty well.

Kathrine

So in your experience, what accessibility problems do developers often overlook, and where can AI tools help?

Professor Wu

Yeah, I think, so in terms of accessibility, I think there's one of the biggest problems I think have, like, a really, like, have really big potential to improve is just awareness. So, like, sometimes when you tell people that a blind person can use a computer at all, they're actually pretty surprised. Like, oh, wow, how do they, like, click the buttons?

Like, how do they know, like, they're actually surprised that they can use it at all? And so they don't know about accessibility, or they don't know about, like, guidelines, or the existence of accessibility technology. So that's, like, one step of it, and there are a lot of AI tools out there that can, and I've worked on some of them, that can, like, look at something you've made, and then sort of take, like, a general set of guidelines and say, like, hey, did it meet the right color contrast requirements, or things, like, big enough to be tapped, usually for most people.

Things like that are one way. I think another way that I'm pretty excited about is the way of kind of giving a little bit more nuanced or, like, simulated feedback. So I think the gold standard is to take your, whatever you built as a developer, and then test it with a wide group of people, people with disabilities, people who use accessibility technology.

But the problem is, like, if you do that too soon, you might end up with kind of, like, almost wasting people's time. Like, if you have, like, really basic errors, like, none of the images are described, or, like, it won't even load in a screen reader software, then you're, like, really missing a lot of low-hanging fruit, and the person you're testing with probably won't give you, like, the really nuanced, you know, great feedback that you want. So a lot of the work we're trying to do is sort of build models that will, like, you can kind of condition it on someone's preferences or, like, a persona, and they will kind of give you simulated feedback in the first place that will kind of help you with those basic areas, and then maybe, like, you would be in a better spot to go test with real people.

So that's one way of using AI for accessibility that I'm pretty excited about and we're working towards.

Kathrine

So you used to work in human-centered AI at Apple before starting the CIDR lab at Purdue. So how has moving from industry to academia changed the questions that you research and your day-to-day work?

Professor Wu

Yeah, so Purdue and Apple, they're, I mean, yeah, they're both really great places to work. I think, like, and actually I'm still working on a lot of the same long-term, you know, problems and trajectories, right, like how do we, what does it mean to quantify design, how do we apply computational approaches, and so on. I think one big difference is where you get inspiration for problems from.

So at a company like Apple or any company, we, you know, those companies make really great and impactful products, like a lot of people are using them, and we're constantly thinking about, like, well, how might we make this product or service better for our users, or maybe users will even give you feedback, like, oh, you know, I wish, you know, Keynote or whatever, this thing could do this, and so we can kind of think of that way, like, well, how should we address this from a research standpoint, and that's been great and really fruitful for me.

Another way, I think, in academia is, you know, as a professor, you work with a lot of students, and this is not only in the classes that you teach, but also students that you advise. So if you have a lab of graduate students, they are all interested in different things, they have different ways of getting ideas, so you're almost, like, you get your inspiration from them in terms of, like, what they're excited about, what you end up advising them on, so I think it tends to lead to actually a pretty broad view of, you know, what you would find interesting or what you end up working on. So I think that's one of the big differences, and, you know, I think I had a great time doing both, but yeah, like, so far, I would say, since joining Purdue as a professor, I've started doing some new projects that maybe I wouldn't have done if I've only been thinking about, you know, products of a certain company.

Kathrine

And finally, what is one realistic thing a high school student could do this summer to begin exploring human-computer interactions?

Professor Wu

So I think the cool thing about human-computer interaction is it studies how, you know, humans interact with computers, and, you know, presumably we're all human, and we use a lot of computers, right? So in some sense, right, like, there's this term that gets thrown around sometimes, like, first-person research. We are all users, and we are all using, like, computers all the time.

We can, like, if we kind of adopt a very, like, scientific or critical mindset and notice things, right? Like, you know, why does, you know, X program do this? Wouldn't it be great if it did something else?

And then, so, like, that's the mindset. The other thing is, like, I think it's really cool that the barrier to implementing a lot of things have been decreased a lot with, like, AI tools, right? So, like, why does Google do this?

Well, maybe you can make, like, an extension that makes it so that what if it didn't do that, or, you know, like, these different things. I think those are something that you can kind of engage. That's one way of getting involved in building these systems and features that underpin a lot of HCI research.

And I think in terms of HCI, like, as a field versus HCI as a discipline, or sorry, like, as a research area, if you're interested in research, it would also be a really good skill to start looking for published academic literature in that area. And so, the good news is AI has also made finding papers and understanding papers a lot easier. So, you know, if you had some questions that you thought about, you could definitely ask these tools, or even search for yourself on, like, Google Scholar or something, tools that have been, you know, in the similar area.

And then that can kind of get you interested in the research side of things, like, who else has been thinking about this problem? Is there a deeper theoretical underpinning in all of these things? I would say, like, learning to read a paper yourself is, like, without AI summarization, is also an extremely important skill, because eventually you would want to learn how to write and communicate effectively.

But, yeah, like, as a, you know, something that you start out with, I think it's a great tool to kind of get exposed to what's out there. So, yeah, so I guess to summarize, like, look for problems, like, you know, in your everyday use as, like, a first-person user. Think about, like, how you might be able to solve those problems.

Look for related things that people have tried. And it's just kind of shifting mindsets from, like, only a peer user to actually someone who's, like, kind of looking for ways to improve.

Kathrine

All right, so that's a great place to end. Thank you so much for sharing your work and your advice with me. I really appreciated it, and I think students will learn a lot from hearing your perspective.

Professor Wu

That's great. Yeah, thanks for having me.