Data Mine Students to Measure Grit?

Posted Truth in American Education on April 2, 2013


The U.S. Department of Education’s Office of Educational Technology published a report back in February entitled “Promoting Grit, Tenacity and Perseverance: Critical Factors for Success in the 21st Century.”  From the executive summary:

How can we best prepare children and adolescents to thrive in the 21st century—an era of achievement gaps that must be closed for the benefit of everyone in society, rapidly evolving technology, demanding and collaborative STEM knowledge work, changing workforce needs, and economic volatility? The test score accountability movement and conventional educational approaches tend to focus on intellectual aspects of success, such as content knowledge. However, this is not sufficient. If students are to achieve their full potential, they must have opportunities to engage and develop a much richer set of skills. There is a growing movement to explore the potential of the “noncognitive” factors—attributes, dispositions, social skills, attitudes, and intrapersonal resources, independent of intellectual ability—that high-achieving individuals draw upon to accomplish success, (pg. v).

When I think of “grit,” I think of John Wayne in True Grit, and in a way they refer to something similar as well.

They define “grit” as, “Perseverance to accomplish long-term or higher-order goals in the face of challenges and setbacks, engaging the student’s psychological resources, such as their academic mindsets, effortful control, and strategies and tactics,” (pg. vii).

“Grit,” I believe is a personality trait and is mostly the result of someone’s upbringing.  To the Feds, however, it’s something that must be measured, data mined and shared.

There are many different types of measurement methods, each with important tradeoffs.

  • Self-report methods typically ask participants to respond to a set of questions about their perceptions, attitudes, goals, emotions, beliefs, and so on. Advantages are that they are easy to administer and can yield scores that are easy to interpret. Disadvantages are that people are not always valid assessors of their own skills, and self-reports can be intrusive for evaluating participants’ in-the-moment perceptions during tasks.
  • Informant reports are made by teachers, parents, or other observers. Advantages are that they can sidestep inherent biases of self-report and provide valuable data about learning processes. The main disadvantage is that these measures can often be highly resource intensive—especially if they require training observers, time to complete extensive observations, and coding videos or field notes.
  • School records can provide important indicators of perseverance over time (e.g., attendance, grades, test scores, discipline problems) across large and diverse student samples. Advantages are the capacity to identify students who are struggling to persevere and new possibilities for rich longitudinal research. Disadvantages are that these records themselves do not provide rich information about individuals’ experiences and nuances within learning environments that may have contributed to the outcomes reported in records.
  • Behavioral task performance measures within digital learning environments can capture indicators of persistence or giving up. Advantages are that new methods can be seamlessly integrated into the learning environment and provide unprecedented opportunities for adaptivity and personalized learning. Disadvantages are that these methods are still new and require intensive resources to develop, (pg. ix-x).

Pages 35-49 in the report give further detail in the type of measurements that would be deemed helpful.  One suggestion they give with self-report entails students to carry a device with them to report their feelings during certain situations:

For example, researchers can examine consistency in participant’s ratings to determine the strength of the belief or skill. Self-report can also be used to measure process constructs; for example, in the Experience Sampling Method (ESM), participants typically carry around a handheld device that “beeps” them at random intervals, prompting self-report of experiences in that moment (e.g., Hektner, Schmidt, & Csikszentmihalyi, 2007). Such data can be used to make inferences about emotions, thoughts, and behaviors within and across specific situations, (pg. 35).

Below is example of a self-report measure of “grit.”


Then you have informant reports, and they cite something that KIPP is doing with their students:

For example, KIPP and other character education programs have been developing methods of using explicit teacher feedback to help students gauge their level of grit with respect to specific criteria and to open up conversations among parents, teachers, and students (see Chapter 4 for more details about these models). These schools have been implementing a “Character Report Card” on which students receive ratings pooled from multiple teachers on factors such as grit and self-control. Exhibit 9 illustrates what such a report card might look like. It is important that these ratings come from multiple teachers, as they are then less susceptible to biases of particular relationships. KIPP has been facilitating the use of these Report Cards with a technology called PowerTeacher that allows teachers to input their ratings online. Informant reporting is also a common approach for teachers, parents/guardians, and mental health professionals to assess the social-emotional competencies that serve as protective factors associated with resilience in young children. Forexample, the Devereux Student Strengths Assessment (DESSA; LeBuffe, Shapiro & Naglieri, 2009) can be used for children in kindergarten through eighth grade (ages 5-­‐14). The DESSA is a 72-­‐item, standardized, norm-­‐referenced behavior rating scale that focuses on student strengths and positive behaviors related to eight dimensions: self-­‐awareness, social awareness, self-­management, goal-directed behavior, relationship skills, personal responsibility, decision making, and optimistic thinking. It can be used for screening, profiling for intervention, and monitoring and measuring change (Hall, 2010), (pg. 38).

Below is an example of a character report card from KIPP:


Then you have record keeping.  They tout the example of a Youth Data Archive at the Gardner Center at Stanford University.  This archive, “links data across systems—school, social services, foster care, youth development programming, juvenile justice—to provide actionable integrated student profiles to educators,” (pg. 40).

Having formerly worked in the juvenile justice field I have a HUGE problem with any data gathered in the juvenile justice system with anyone not directly working in that system.

Why do educators need an “actionable integrated student profile”?  So they can place a label on a student?  Are kids and parents giving consent for any of this information to be shared?

Then there is “behavioral task performance.”

While laboratory experiments have examined behavioral task performance for many years, new technological opportunities offer potential for new methods and approaches. Educational data mining (EDM) and learning analytics within digital learning environments allow for “micro-level” analyses of moment-by-moment learning processes.

Student data collected in online learning systems can be used to develop models about processes associated with grit, which then can be used, for example, to design interventions or adaptations to a learning system to promote desirable behaviors. Dependent behavioral variables associated with a challenge at hand may include responses to failure (e.g., time on task, help-seeking, revisiting a problem, gaming the system, number of attempts to solve a problem, use of hints), robustness of strategy use (e.g., planning, monitoring, tools used, number of solutions tried, use of time), level of challenge of self-selected tasks, or delay of gratification or impulse control in the face of an enticing off-task stimulus. Such data can be examined for discrete tasks or aggregated over many tasks, (pg. 41).

They also discuss affective computing (the study and development of systems and devices that can recognize, interpret, process, and simulate aspects of human affect.)

Researchers are exploring how to gather complex affective data and generate meaningful and usable information to feed back to learners, teachers, researchers, and the technology itself. Connections to neuroscience are also beginning to emerge, (pg. 41 emphasis mine).

President Obama just today called for investing $100 million for a brain mapping project.  USA Today reports that it’s goal is to develop “new technologies that can record the activities of individual cells and neurons within the brain.”

They then devote a short paragraph on the ethics of garnering this new type of personal data.

As new forms of measurement emerge and new types of personal data become available, the field must also deal with critical ethical considerations. Of course, privacy is always a concern, especially when leveraging data available in the “cloud” that users may or may not be aware is being mined. However, another emergent concern is the consequences of using new types of personal data in new ways. Learners and educators have the potential to get forms of feedback about their behaviors, emotions, physiological responses, and cognitive processes that have never been available before. Measurement developers must carefully consider the impacts of releasing such data, sometimes of a sensitive nature, and incorporate feedback mechanisms that are valuable, respectful, and serve to support productive mindsets, (pg. 48).

They recognize that users may not be aware they are mining data.

Let’s focus on schools being places where students learn instead of being a laboratory where student’s behavior can be studied, recorded and shared.  How much of this mentality will be present when SBAC and PARCC assessments (which will be used to mine data, we are just not sure of the extent) are released remains to be seen.

“Grit” isn’t something that schools will be able to “can” and give out to students so measuring it seems rather ludicrous to me, especially at the expense of privacy.

Shane Vander Hart is the Editor-in-Chief of Caffeinated Thoughts, a popular Christian conservative blog in Iowa. He is also the President of 4:15 Communications, a social media & communications consulting/management firm, along with serving as the communications director for American Principles Project’s Preserve Innocence Initiative.  Prior to this Shane spent 20 years in youth ministry serving in church, parachurch, and school settings.  He has taught Jr. High History along with being the Dean of Students for Christian school in Indiana.  Shane and his wife home school their three teenage children and have done so since the beginning.   He has recently been recognized by Campaigns & Elections Magazine as one of the top political influencers in Iowa. Shane and his family reside near Des Moines, IA.  You can connect with Shane on Facebook, follow him on Twitter or connect with him on Google +.

Is The U.S. Dept. of Education Violating Federal Law by Directing Standards, Tests & Curricula?

Posted on February 11, 2012 by truthed

BOSTON, MA —Despite three federal laws that prohibit federal departments or agencies from directing, supervising or controlling elementary and secondary school curricula, programs of instruction and instructional materials, the U.S. Department of Education (“Department”) has placed the nation on the road to a national curriculum, according to a new report written by a former general counsel and former deputy general counsel of the United States Department of Education.

The Road to a National Curriculum: The Legal Aspects of the Common Core Standards, Race to the Top, and Conditional Waivers  (also embedded below) is sponsored by Pioneer Institute, the Federalist Society, the American Principles Project, and the Pacific Research Institute of California.

With only minor exceptions, the General Education Provisions Act, the Department of Education Organization Act, and the Elementary and Secondary Education Act of 1965, as amended by the No Child Left Behind Act (NCLB), ban the Department from directing, supervising, or controlling elementary and secondary school curriculum, programs of instruction, and instructional materials.

“The Department has designed a system of discretionary grants and conditional waivers that effectively herds states into accepting specific standards and assessments favored by the Department,” said Robert S. Eitel, who co-authored the report with Kent D. Talbert.

The authors find that the Obama administration has used the Race to the Top Fund and the Race to the Top Assessment Program to push states to adopt standards and assessments that are substantially the same across nearly all states. “By leveraging funds through its Race to the Top Fund and the Race to the Top Assessment Program, the Department has accelerated the adoption and implementation of the Common Core State Standards (“CCSS”) in English language arts and mathematics, as well as the development of common assessments based on those standards,” added Talbert, former General Counsel of the Department.

Through its Race to the Top Assessment Program, the authors explain how the Department has awarded $362 million to consortia to develop common assessments and measure student achievement.  Two consortia have won a total of $330 million, and each has been awarded an additional $15.9 million supplemental grant to “help” states move to common standards and assessments.

“There is no constitutional or statutory basis for national standards, national assessments, or national curricula,” said Bill Evers, research fellow at Stanford University’s Hoover Institution and Koret Task Force on K-12 Education member. “;The two testing consortia funded by the U.S. Department of Education have already expanded their activities well beyond the limits of the law. As this paper recommends, the actions of the Department warrant congressional hearings.”;

One of the consortia has stated directly that it intends to use these federal funds to support curriculum materials and that it expects to use the money to create a “model curriculum” and instructional materials “aligned with the CCSS” pushed by the Race to the Top Fund.  Secretary Arne Duncan has said that the work of the two consortia includes “developing curriculum frameworks” and “instructional modules.”

“Frankly, this makes sense,” said Eitel.  “How can one design assessments without taking into account what is taught?  But the legal concern is that these federally funded assessments will ultimately direct the course of elementary and secondary course content across the nation,” Eitel added.  “This raises a fundamental question of whether the Department is exceeding its statutory boundaries,” Talbert said.

“Proponents of national standards, curriculum and tests claim they’re merely a logical extension of previous federal education initiatives,”said Pioneer Institute Executive Director Jim Stergios.  “The key difference is that prior to Race to the Top and the recently announced federal waivers, the US Department of Education abided by statutes explicitly prohibiting federal direction, supervision, or control of curricula or instruction.”;

The authors also explore how the Department’s NCLB conditional waiver program, announced last September, is driving the states toward a national K-12 curriculum and course content.   To obtain a waiver from the Department, each state must declare whether it has “adopted college- and career-ready standards” in reading/language arts and mathematics “that are common to a significant number of States.”  States seeking waivers must also declare whether they are participating “in one of two State consortia that received a grant under the Race to the Top competition.”  The Common Core State Standards and the assessments consortia are effectively the only ones that fit these descriptions.

“Our greatest concern arises from the Department’s decision to cement the use of the Common Core State Standards and assessment consortia through conditional waivers,” said Eitel.  “The waiver authority granted by Congress in No Child Left Behind does not permit the Secretary to gut NCLB wholesale and impose these conditions,” added Talbert. “As shown by the eleven states that have already applied for waivers, most states will accept the Common Core State Standards and the assessment conditions in order to get waivers,” Talbert stated.

States need not apply for waivers, the authors said, but most states are desperate enough to escape No Child Left Behind to agree to the conditions.  “And once a state receives a waiver, escapes NCLB’s strict accountability requirements, and makes the heavy investments required by the standards, that state will do whatever it takes to keep its coveted waiver,” said Eitel.  In the view of the authors, these efforts will necessarily result in a de facto national curriculum and instructional materials effectively supervised, directed, or controlled by the Department through the NCLB waiver process.

In their analysis, Eitel and Talbert propose several recommendations, including the enactment of legislation clarifying that the Department cannot impose conditions under its waiver authority, as well as congressional hearings on Race to the Top and waivers to ascertain the Department’s compliance with federal law.

Pioneer Institute led a campaign in 2010 to oppose the adoption of national standards, producing a four-part series reviewing evolving drafts.The reports compared them with existing Massachusetts and California standards, and found that the federal versions contained weaker content in both ELA and math. The reports, listed below, were authored by curriculum experts R. James Milgram, emeritus professor of mathematics at Stanford University; Sandra Stotsky, former Massachusetts Board of Education member and University of ArkansasProfessor; and Ze’ev Wurman, a Silicon Valley executive who helped develop California’s education standards and assessments.