Dr. Povey is a renowned research scientist in the field of speech recognition. Over the past twenty years or so, Dr. Povey has been constantly making magnificent contributions to the speech recognition community, making him one of the best known researchers in the field.

Dr. Povey is first known for his work on discriminative training of hidden Markov model-based speech recognition systems. The improved training methods proposed by Dr. Povey were among the most effective techniques for improving speech recognition systems at the time.

Dr. Povey is probably best known for his work on the open source speech recognition toolkit Kaldi. Kaldi was a byproduct of the 2009 summer workshop at the Center for Language and Speech Processing, the Johns Hopkins University, titled “Low Development Cost, High Quality Speech Recognition for New Languages and Domains”. Kaldi was officially released on May 14th 2011, and quickly became the best open source speech recognition toolkit. Since its initial release, Kaldi has been maintained and developed to a large extent by Dr. Povey.

The screenshot below shows the Google Scholar citations of Dr. Povey’s papers.

To better develop Kaldi, Dr. Povey took a faculty job at the Center for Language and Speech Processing, the Johns Hopkins University since 2012, and has been devoting his time purely to developing Kaldi and advising students since then, until the recent termination by the Johns Hopkins University.

A lot of people know Dr. Povey’s work on speech recognition, but only a few know him personally. There is this article on Zhihu (China’s version of Quora) written by Dr. Povey’s former student, which gives a better view of what kind of a person Dr. Povey really is. The article has been upvoted by thousands of readers. The original article was written in Chinese, and below we’ve tried our best to translate that into English.

The Zhihu Article

I’ve never seriously written an answer on Zhihu before. But this time since the topic relates to my advisor, I’ve decided to finally provide a carefully written answer. I’ll leave my name up there.

For the record: I am Dan’s first Ph.D. graduate.

The most upvoted anonymous answer has provided a thorough description of the incident, which is consistent with what I’ve learned, so I’m not going to elaborate on the incident itself again. I’d like to write about Dan, Dan as a researcher, as an advisor, and as an ordinary person. I believe after learning more about who Dan is, people will likely have a better understanding of what really happened during the incident. I’d also like to take the chance to savor some of my Ph.D. memories..

Workaholic

For those people who have used Kaldi, most likely you also know that Dan is a workaholic. Dan’s family lives on the West Coast, but to better develop Kaldi, Dan spent 7 years in Baltimore (yes, I just realized it’s been 7 years!). Dan spend almost all his time developing Kaldi every day, except for eating and sleeping. Of course, sometimes he also sings a song, or plays his electric piano. I believe he works more than 14 hours a day.

When I first met Dan, Dan said “send me emails whenever you have questions, I’m almost always online.” Frankly I didn’t really take it seriously in the first place. Who would always be online? But I was quickly proven wrong. Most of my emails sent to Dan were replied within 5 minutes, throughout my entire Ph.D., no matter if it was on a weekday or on weekends. Same happened to the emails sent to the Kaldi forum. Dan would reply to almost all the forum emails (mostly asking for helps) as long as he was up and working. It was quite amazing that Dan was able to process all those emails efficiently. I started doing open source projects by myself later on, but I was never able to reply to all the emails.

While I was at Hopkins, a majority part of Dan’s work was to write Kaldi code. I just checked on Kaldi’s GitHub repository and by the time I checked Dan had already contributed millions of code to Kaldi, which does not include code that was written but was not submitted. Sometimes when we felt like we should finish something soon, we would sit in Dan’s office and write code together with Dan. There was a time when I was writing code in Dan’s office, Dan’s daughter skyped him to show him her latest paintings. Dan gave a few comments, said “Dad needs to get back to work.” He then closed Skype and started writing code like crazy again…

But of course it is not just about diligent. Dan is very talented. From time to time, Dan would give ideas that sounded mediocre in the first place, but turned out to be very effective after implementation. I often feel that Dan was born for the field of speech recognition.

Open source dream

Oftentimes, when asked about what he really wants to achieve, Dan would most likely say something like reducing the word error rate (a metric for speech recognition systems). But if you have a couple of drinks with Dan and Sanjeev, there is a chance that he would tell you that he would like to replace Nuance (a company famous for its commercial speech recognition products) and provide speech recognition tools for free.

Dan likes to share. Whenever there is a new dataset, or a new algorithm, Dan would open source them as soon as possible, so that people can try them for free. Frankly speaking, Dan can easily commercialize some of the datasets or algorithms, or find a job with much higher salary, if he would like to. But Dan made his decision to stay in Baltimore and develop Kaldi.

I feel Dan really likes the field of speech recognition, and he would love to see people using speech recognition technologies for free.

Willing to help

At first glance, Dan may not seem to be a person who is willing to help others: he doesn’t talk much and he is always focusing on his own work. But deep down inside, Dan is a very kind person, and is willing to help.

One of my friend once told me a story of Dan sharing a room with him. In the early days of Kaldi’s development, researchers would get together in the summer in Czech Republic and have Kaldi workshops. My friend joined one of the workshops, but he didn’t have any funding supporting him. Dan learned the situation, and offered to share his room with my friend. He even covered my friend’s daily cost, out of his own pocket.

CLSP’s (Language Speech Processing Center) master program is another example. For quite a while, CLSP was trying to set up a master program on language and speech technologies. The master program would increase CLSP’s impact on industry, and would also bring in more funding for general research. The process, however, was lengthy and full of hassle, and was hanging there for quite a while without any progress. Dan took the responsibility and pushed the program forward, including designing the curriculum, getting approvals from the university, filming the promotional film, etc.

And also, the CLSP server administrator job. CLSP had an IT administrator called Carl. Over the years CLSP purchased more and more server machines, and finally Carl was not able to handle all the server issues (such as overheating and restarting) alone. Dan then offered to share the responsibilities of the IT administrator job. Whenever there was a problem with the server, Dan would come in and fix the machines, no matter if it was day or night. It was unfortunate that this IT administrator role led to the incident. And I do believe that Dan was trying to save the data and the server machines throughout the incident.

Being honest

To the best of my knowledge, I think Dan is an honest person, and will not lie. There was a very talented junior student who was smart, fast, but was also a little bit lazy. For example, I had to be very creative to make him work while I was working with him. But both Dan and me liked this student. When this student was looking for a job upon graduation, he asked Dan to write a recommendation letter for him. Dan agreed, but also warned the student that he would be very honest in this letter, and it might be a little bit risky. The letter, turned out, almost cost the student’s job. The company asked for an additional letter, and it was Sanjeev’s additional explanation that saved the student’s job.

Dan is not a person who will lie, so I’m inclined to believe what he said regarding the incident.

Sticking to principles

In general people all have their own principles that they are living up to, and Dan is no exception. Dan is a nice and shy person, but there are things that are unacceptable to Dan. Based on my observations over the years, among those unacceptable things are time wasting and server abusing.

Dan hates his time being wasted. If possible, he would avoid most of the social activities, and keep himself in the office working. Couple of years ago, we had a Babel program meeting at a hotel close to Washington DC. By the time the meeting finished, it was around 3PM, and the traffic started to kick in. People however were still hanging around there connecting with each other. Dan was really worried that he would be caught in a traffic jam, so he persuaded me to quit the social part, and drive together with him back to Hopkins. He even offered to drive because I was driving too slow… Dan drove all the way back to Hopkins, and started working again after getting back to his office.

Dan also hates CLSP’s server being abused. Dan is nice to most of the things, but if someone abuses the CLSP server, they would receive a stern warning from Dan. Dan feels that data and computing resources are both very important assets to CLSP students and faculty members, and they should be available to CLSP at all times. This probably explains why Dan tried to retake the server room in this incident.

Postscript

Dan’s work and contribution in speech recognition is well known, so I believe that this incident likely will not have much impact on his career, nor will it have much impact on Kaldi’s development (as far as I know, his next company and manager support open source). However, it will deeply hurt the speech recognition development at CLSP and Hopkins. The outcome of this incident is really unfortunate. I hope that the university will come up with a fair and transparent investigation result, and that CLSP and Hopkins will weather through this storm. I’m attaching an email I sent out to the group earlier today to close my answer: