Posts Tagged ‘analytics’

In which an incident is deconstructed and the impressions received and left behind might be more important than what may or may not have actually happened.

So let me be clear from the start: No laws were broken. As far as I can tell, every single requirement and guideline for the ethical use of human subjects was met, at the very least on paper and maybe even, more or less, in person. The sense of unease stems not from deliberate actions (probably), but more due to a potential neglect of questionable benign-ness, or perhaps neatly convenient acts of omission.

It’s just that contextually, based on perhaps singular perceptions and a long history of interest in personal privacy in the digital age, it all seems kind of… sneaky.

And while what follows isn’t really about learning analytics, I can’t help but think that it could be, someday. Full disclosure and voluntary participation are all well and good on paper. But do real-life contexts and circumstances really encourage or allow people to understand the nature and use of the personal data they are casting off as they travel through life… or through educational institutions?

Tensions on the fringe

One can dodge and weave and innovate in learning and education all one wants, but as no innovation or innovator is an island, there regularly appear those moments and phases when one bows, however temporarily, to convention. This includes, among other things, interactions with the Behemoths of Higher Education.

As our household is currently negotiating compromise phases in which, due mostly to issues of timing, some things which may be useful are still locked behind the gates and in the ivory towers, there are, in the concession to convention, ample opportunities to roll over and play dead on issues which otherwise and previously have elicited independent thought. The single, not-dead-yet issue to be addressed here, lover as I am of the revelation of small things, is a small portion of the small segment of higher education adventures known as first-year-student orientation.

Having been a witness/eavesdropper/story listener for two or three and half such events (lack of convention accounts for the odd numbering) over the past year or two (ditto), it is quite clear that these orientations are largely a) tiresomely mandatory and mass produced and b) much more for the convenience of the institution than the student. Everybody in one place (physically), given (verbally and in reams of print, whether it exists digitally elsewhere or not) the same information (relevant to the individual student or not) at the same time.

Quite a shock to more independent-minded learners, but one must, I have intoned parentally, persevere for the potentially greater good. So to the idea of being lead en masse through a university library tour (ignoring the detailed map at the entrance door) and pointed to such fascinations as “computers that contain databases,” I have said: March on. To the apparently ubiquitous indoctrinal lectures on Why Higher Education is Morally Superior to Other Forms of Existence (OK, I may have renamed that a bit), I have said: Mental and written critiques are appropriate (moral) responses. To the “Maintaining a Four Year Graduation Plan” with matrixed requirement charts, I have rolled my eyes.

And to the late-afternoon text from the depths of one such orientation that said “They’re asking me about my religion and your income and if I have ever consumed alcohol or believe in abortion. I’m pretty sure they can’t REALLY ask that… can they?” …. I said “Walruses Tell Fables!?” (or something similar).

Yes, They can: The Freshman Survey

As it turns out, They indeed can ask these things. They do ask these things. And, over the course of the next 15 minutes on the side of the bike trail with mobile device in hand and the guidance of the mobile device and descriptive narrative at the other end of the texts, we discovered that They’ve been doing so since 1966.

Somehow, the Cooperative Institutional Research Program Freshman Survey had escaped my notice. I don’t remember taking it (which may be significant in and of itself), and the other Young College Person in the household has for some reason never seen one. But there it is. A massive trove of highly specific and personal data, gathered from eight and a half million students over 40-plus years.  Researched and cited six hundred ways from Sunday over the years (and processed at the current rate of a baseline fee of $775 per institution plus $3 per survey by the Higher Education Research Institute ). This is the Godzilla of youth surveys, a socio-demographic rainbows-and-ponies dream source, with 200,000 students surveyed in 2011 alone.

Even if one were inclined to tilt at windmills and dragons, to criticize this survey and its behemoth backers or its makers or responders or interpreters or yes, its beneficiaries (because one must stand in awe of the sheer massiveness of the data) would be the height of wasted effort. And to clarify: In the big-picture-well-after-the-fact of the actual “survey incident,” let me be clear that the Higher Education Research Institute’s website pertaining to this survey exhaustively addresses the processes and procedures and ethics statements in ways that fulfill all of the necessary categories of due diligence and non-coercion and sincere expressions of desire to gain valid data.

In other words, they have — in print and in public— clearly covered their hindquarters on all of the concerns that arose in that 15-minute window on an early June afternoon. They are untouchable in the world of academic and scientific freedom and responsibility. And no, students do not HAVE to answer the questions. All, some, or none—totally voluntary. Except…

It’s the context beyond the content

There remains this fascinating and disturbing gap, or chasm, or Grand Canyon of Disconnect, between what happens with the ideally ethical survey plan with its detailed CYA statements on a website, and the experience of the (or this particular) surveyed student on the other end of the texts with a folder containing a four-page bubble test and a single page of explanatory fine print. So context becomes critical. How to pick this apart…?

Well, let’s start with the setting and lead-up to the survey administration. At the University of Massiveness, students are corralled in some-hundred batches for 30-some hours and trotted through icebreakers, tours, course advisories, registration processes, and sessions in financial and personal responsibility, etc. Lather, rinse, repeat throughout the summer, guided over multiple weeks by Ernest Older Students/camp counselors.

If I recall the half-hour I spent as a parent in the Welcome speech correctly (and again, impression is greater than reality here), the idea of the survey was introduced or at least referred to in passing by an administrator at the very beginning of the day. How Official. And the Freshman Survey is listed on the day’s events, printed clearly and capitalized (how official) in the multi-colored, high-quality (official) orientation booklet, to take place in a rotation with Dinner and Dorm check-in, depending one’s subgroup. And so by the time the actual surveying occurs, new students have been receiving a day-long stream of authority-based messages about expectations for university students. And, as my texter noted, having been “talked at” for several hours, patience and attentiveness and the ability to absorb any more information was wearing very thin. So here’s where things get tricky.  The HERI website clearly states:

If your campus administers the survey in a proctored on-campus setting, the following text should be read aloud to students prior to distributing the instruments. While you are free to personalize or edit this text, the points in bold must be included:

“We ask that you complete this survey as part of a national study of college students conducted by the Higher Education Research Institute at UCLA. One major goal of this research is to determine what happens to students when they attend college. This study is designed to help improve the quality of the college experience and thus may benefit future generations of students. Results of your participation also will be directly beneficial to students at [name of your campus], since we will receive complete tabulations of your responses to compare with the responses of students at other institutions. We ask for your name and address so that the researchers at UCLA can contact you at some later date for a follow-up study. We would also like your Student ID Number so we can merge your responses with other campus data to support our institutional research program. You also may complete this survey on the Internet—please refer to the attached instructions if you wish to do so. Your responses will be used only for research purposes and will be strictly confidential. Please read the ‘CIRP Freshman Survey Information Sheet’ for more information about your rights as a participant in this research. Your participation is voluntary and will not affect your standing at [name of your campus].” 

And here’s what I know from the texts I received and the subsequent conversation upon arrival home: the texter has absolutely NO recall that that this statement was read. True, we can’t say that it was or it wasn’t. But after six or seven hours of information and the promise of dinner just around the corner, how many of the students were cognitively equipped to hear and understand the message and its implications even if it was read?

And yes, you read that right:

“We ask for your name and address … [and] your Student ID Number…” (CYA website disclosure says: “Before receiving any responses your school is required to certify in advance that the data will only be used for research purposes will not be used to investigate specific individuals.”  Um, so California can call you personally, but your local school promises not to? And what about that loaded word…”investigate”?)

Familiarity and disorientation

And what do we know about these named and numbered (but we’ll hide our eyes, really we will) first-year students, survey or not? Well, I’d surmise that most got there by playing the Educational Accomplishment game of tests and coursework well enough to be accepted as one of the eight applicants for every seat—that was an important point made during the Welcome speech. We also know that these students are the direct legacy of the No Child Left Behind era, where testing has been of huge importance, from yearly progress measurements to the absolutism of the AP testing sold as the pinnacle of success.

Most of them have not been away from home for very long (many, based on a show of hands, were looking at life in the “big city” for the first time and over half of them had never done their own laundry), and many of the parents were just across campus on their own “orientation” track, where they were, based on the schedule, being reassured about the university’s ability to act in loco parentis. One could also surmise that most students were about 18 years old and a week or two out of high school.

The reason I detail this is that while I have a strong belief in and first-hand knowledge of the potential of young people to think critically and innovatively when sincerely offered the necessary freedoms and opportunities to do so, I can’t help but think that none of the characteristics noted here really set up these young students to critically reflect on what they were being asked to do. After a steady cultural diet of “college is the ultimate goal,” with the “college selection process” fraught with extreme family and school anxiety and then focusing on new-found loyalty to mascots and colors and the commitment of great sums of money… well, certainly it is preferable to believe that anything your college feeds you is delicious and nutritious. Including the survey in its official time slot.

And let’s talk about the survey administration method “preferred” by HERI:

“The best results occur when the survey is administered in a proctored setting.”

Which means, like any “test,” in a formal setting. In a classroom. With someone “in charge” watching you. And other students lined up next to you in their desks. The texter later noted she “felt like” They said something about students not being able to leave until they had completed the survey, but then realized They could just as well have said, “You can leave when you are done,” in which “done” (in other words, a completed survey) could be inferred as a necessary requirement for departure.

And the survey itself? In this case, a fill-in-the-bubble multiple choice, just like every other Very Important and Official test given to students to measure their yearly progress beginning in kindergarten or to assess their college readiness. I think one can’t underestimate the Pavlovian response “high-achieving” students such as those accepted by the University of Massiveness might have to another multiple choice test.

It must feel very familiar. And mandatory. Which might explain an astonishing 90 percent-plus national completion rate, if I understand correctly.

And finally, there was the content of the survey itself… which is where the eyebrows began to rise in the first place.

Oversharing… or oversensitivity?

Here are some questions from the survey… (and some cantankerous interpretations):

What is your best estimate of your parents’ total income last year? (Report on your parents’ personal information without their knowledge.)

For the activities listed below, indicate which ones you did during the past year [by frequency]?

“drank beer/”wine or liquor” (Would you like to admit to doing something illegal?)

“was a guest in a teacher’s home” (Just… creepy)

”Failed to complete homework on time/“came late to class”/ “fell asleep during class” (How compliant are you and can we really expect much of you?)

Also included: political litmus ratings scales for issues such as:


Death penalty

Racial discrimination

Taxing the wealthy

Same-sex marriage

Free speech

And questions relating to mental health, with references to depression, emotional health, self-understanding, and counseling.

Not to mention questions that initially seem truly a shot in the dark:  personal opinions about future plans regarding degrees, life goals, philosophical views, the nature of success, etc. Why ask such nebulous, non-verifiable things? Because wait, there’s more. HERI has THREE MORE SURVEYS (not that those new, proctored students know this) that can be administered during a student’s college career. And all of a sudden the massiveness and cross-validation of the data on both the cohort and each individual takes on proportions the mind simply fails to grasp.

True, it might not be “worth it” to care all that much. Between Facebook and financial aid forms, a lot of the survey info could be obtained otherwise, although with much less ease.

Are the survey responses interesting and valuable as a picture of society in evolution, among other things? Yes, yes of course. Is there anything necessarily bad about sharing this kind of info? Maybe not. But is it really something people want to put out there with the assumption that it will forever and ever be treated benevolently by the universe? Would this data make great political fodder when the database is hacked or leaked and the responses from individual students are plumbed? Oh. My. And if cooperative students fill out surveys like this at regular intervals, are they fully equipped to grasp the implications of this first step going forward when they are at the end of a long day of potentially disorienting orientation, even if the “bigger picture” were highlighted for them?

Keeping the uncooperative ten percent at ten

It is the Great Trade-Off. As a researcher or an institution, data is gold, and everyone can sleep better by not scrutinizing too closely any issues of context if you’ve otherwise met all of the research ethics guidelines. It’s just easiest to deliver this survey, sanctioned by all sorts of Powers that Be over time as it is, without highlighting any of this messiness, isn’t it?  The extraction of personal data is made incidental and relegated to background noise—just another “thing” in the course of the day. And as the survey subject, well, just close your eyes, and it will soon be all over, and unless you think about it (or the Queen, perhaps), you’ll never know or recognize exactly what you’ve given away or revealed. Unless… well, the “unless” is up to your imagination.

And ultimately, why should I, personally, make a fuss of several hundred words? After all, my afternoon texter, who has an apparently unusually acute sense of personal privacy, refused to be seduced into complacency by the Dark Lords of Data (or, optimistically, the Well-Meaning Minions of Measurement). But in this stance, she also had:

1.) The reflective/critical capacity to consider the implications of what was being asked and wonder why.

2.) The curiosity to seek further information by locating and plowing through the  “Survey Information Sheet” (until then ignored in the mass of other paperwork) that revealed the optional-ness of participation.

3.) The ability to use a mobile device to obtain information and consultation outside the immediate context to further validate her questioning and understanding, and

4.) The self-assuredness to walk out of a full room of peers under authoritative supervision without handing in her survey. (At which time her earnest seat neighbor noted she was texting her mother to find out which family income category she should mark.)

I would think higher education researchers in particular would be more interested in understanding the development of these skills and attitudes than in homework and partying habits. But chances are, this particular subject probably would react the same way to a survey about that as well.

One other small point: This intended survey subject/texter happened to be several solstices short of legal age. We generally note that if you’re invited to and intend to hold your own in unorthodox ways, you do it without playing the age card. Age is, however, an issue which is addressed on the HERI website (nothing if not thorough), where it is noted that parental permission may be necessary for minors to participate in the survey. But again, this was not addressed in the immediate context of taking the survey at the University of Massiveness. So the written information (for those who chose to read it) that states “Your participation and return of the questionnaire indicate your consent to participate in the study” seems a bit disingenuous. Especially since there are bubbles to mark on the survey which acknowledge under-18 students. And what about those tired,  18-plus but perhaps cognitively overloaded students who have just been thoroughly oriented/socialized into compliance with university norms? It seems there is no room for second thoughts once that survey leaves their hands…


Read Full Post »

In which I take a quick glance at another open course.

In spite of our best efforts at incorporating public transportation and, in the warmer months, serious commuter bike mileage, I spend an alarming amount of time ferrying offspring hither and yon. This precludes a lot of other activities, including any consistent writing for such things as open courses. But, depending on the traffic level and the weather, the drive time does foster opportunities to muse upon the events and information of the day.

So in musing about my brief, once-over-lightly, tip-of-the-iceberg foray into learning analytics descriptions and commentary, I found myself reflecting on my agreement with Viplav Baxi‘s response to George Siemens’ question about learning analytics critiques: the potential and actualization of learning and knowledge analytics will make our current systems of assessment and other learning processes look like a horse and buggy before the invention of the wheel.

OK, so I’ve rephrased this a bit, caught up as I am in the role of transportation jockey. But I’ve just spent the past week driving a most modern horseless carriage.  And it struck me as I inched along in a fresh three inches of chemically-converted slush that the vehicle comes equipped with a rudimentary form of “learning analytics,” especially in terms of feedback mechanisms. And here I’ve become acutely aware of the persistent discrepancies between the ideal and the reality from the learner/driver end of the equation as the technologies – and maybe even our psychological processes – undergo developmental and adaptive changes.

True, my car and I agree on the big picture. The point is to get from location A to location B in a safe manner while consuming as few fossil fuels as possible. It was ever thus. (Well, at least in this house, since we have never understood why one would drive vehicles insouciantly named after the landscapes they’re destroying… tundra, sequoia, etc.).

But now my car has taken on the role of data provider and driving analyst in ways that the venerable Small Outdoorsy Wagon has never done. It “responds” to my driving through various signals and signs. It’s a bit trying, at least in this initial phase.

For example, in the name of safety, the car is equipped with numerous bells and whistles, and yes, I mean this literally.  The one most perplexing to me is the Mack-truck-in-reverse beeping that occurs when I put this considerably less intimidating vehicle into reverse. BEEP BEEP BEEP BEEP. Every. Damn. Time…. I leave the garage. It’s not even outside the car. This I could see as a reasonable warning to others that the vehicle and a driver of indeterminate skill are on the move. But it’s inside the car. There’s no override. There’s no volume control. And I’m wondering what research has shown that such vast numbers of drivers are so confused about whether they are coming or going while piloting this car that all drivers need to be warned that they’re going backward.

Or how about the orange “slipping tires” warning light that blinks if there are…well, slipping tires? Which is about, oh, every quarter mile with our current road conditions. My peripheral vision is constantly caught by the flash of Threat Level Orange just off to the left, behind the steering wheel. What’s going to happen over time? I’m going to learn to ignore it, I suspect, which probably wasn’t the intent. And as an experienced driver, believe me, I know if I’m spinning my wheels and need to change tactics. Flashing lights at me just increases the number of things vying for my attention under already problematic circumstances.

I’m also feeling a bit ambiguous about the sheer volume of data that is suddenly available to me as a driver. True, dashboards (and I’ll point out my laptop has one, too) through the ages have provided drivers with all sorts of information. Speed being of most interest, I suppose, both when it was hard to come by and now, when it’s hard to keep down. Fuel gauge. Engine temperature. Oil level. Add some trip mileage. A clock. The radio controls.

What strikes me now, however, is that the degree of precision in this information has increased tremendously. True, it’s my choice (or is it?) to react to the data, but I’m finding a digital readout of 54, 55, 56, 57 (oops) to be a more exhaustive and rigid taskmaster than a needle quivering around the 55 mph mark on a dial. This is also true for the second set of feedback mechanisms that have suddenly appeared: the Hybrid System Indicator. All of a sudden, I know not only my exact trip mileage, but also have second-by-second information on battery power. And on how far I can travel on the remaining fuel at the current rate of speed (as if I could maintain that speed in rush hour traffic). And on exactly how many gallons are left in the tank. And on whether I’m pulling from the battery, from the gas, from both, or whether and how much I’m charging the battery (available as a scaled readout or as an animated illustration that reminds me of those movies of blood flowing through the heart chambers). And even more addictive: I can know the average number of miles I’m getting per gallon every single moment, to one place behind the decimal point. I’m not much of a gamer, but we’ve already developed a friendly household competition to see who comes back to the garage with the highest score.

So one question from a learning perspective is: has this information and analysis (provided partly by the car, and partly through my interpretations) somehow changed my behavior or knowledge as a driver? In this “getting to know you period,” I’d say yes. It’s easy, for example, to use the power monitoring to make minor adjustments to the acceleration rate when pulling away from a stop sign, especially if you’ve developed an aversion to seeing the little indicator zoom into the brown (cleverly equaling “yucky”) fossil fuels zone.

But the other thing that concerns me is how much time I spend looking at these gauges, drawn in by the hobgoblins of consistency and accuracy and constant feedback, and the mixed-motive enticements of low fuel consumption/less pollution/economic savings. The speedometer checks are actually more essential, as the quietness of the high tech engine makes it hard to recognize the speed at which I’m travelling; in other words, I have more potentially “useful” information and thus greater potential control over my “results,” but I am receiving fewer environmental cues. (And how much precision does one really need? Do I really need to know that the car’s interior is 67 degrees, and will my driving experience be all that much cozier if I set it to 69? And finding out that my life behind the wheel averages 23 miles per hour? I think I’d have rather not known.)

I’ve also become acutely aware that consistent monitoring and making use of all this information means… less time looking at the road. Paying less attention to the other cars. Pretty much ignoring the scenery. I might have more safety warnings, but my new, information-rich processes aren’t necessarily contributing to more safe or enjoyable procedures. And all of this information and the constant adjustments I make in response create, I can attest, a more mentally fatiguing driving experience. (Something I, three months into a snowy, x-hundred-rush-hour-miles-a-week winter, wouldn’t have thought possible).

So how much of this new wealth of information and responsive feedback will I simply begin to incorporate without this extra refelection over time? To what degree will I assert my autonomy as a driver and simply ignore what I see as bothersome analysis, fuel consumption results be damned?  How soon will all of this be old hat, whereby the constant exposure to the technology will gradually wear me down into unreflective compliance with those digital measuring sticks, and I’ll likely forget the initial dissonance of these changes? And what about the household’s driver-in-training, whose arcane, state-required  “driver’s education” tells new drivers to honk at bicyclists ahead of them as a warning (wtf?) and preaches about the dangers of cell phones behind the wheel, but doesn’t begin to recognize the new cognitive demands of driving such a technically advanced vehicle, with four screens worth of data accessible via a steering wheel control?

On the other side: does this discussion really capture the full potential of– or any reasonable hesitation about– the sophisticated complexity of learning analytics?  It’s more about an interim or introductory stage in driving analytics, to be sure. Already, there are cars that do far more than mine. Some remember, for example, the preferred interior settings of each individual driver. Some, like the Google car, even drive themselves. Ultimately, it’s clear that I’ll be adjusting to the vehicle, not the other way around, which seems indeed to be the most rudimentary of “responsive” systems.

I also recognize that this conversation is still all about driving, and that’s a paradigm problem. I can get pretty excited about 47.7 miles per gallon when I’d gotten used to a (mentally calculated) 28 mpg. But these new numbers, no matter how improved, aren’t a seriously effective response to the larger implications of fossil fuel consumption in a shifting climate. Better mileage is insufficient for the leap we need to make. So I’m hoping this, too, reflects an interim, rather than ultimate, solution. I’d say the very act of driving needs to be scrutinized as well, along with a whole host of other forms of consumption. (Teleportation, anyone?)

And finally, I’d note that the seductive power of the oversimplified analogy can create a misleading but unfortunately persistent picture. So I suspect I’d best spend more time surveying the route maps and take these musings for another drive… and thoughtfully prepare to cross some fancy new bridges as I come to them. 

Read Full Post »