The Growing Need for Assessment and Accountability
The Five Challenges
An Introduction to the Technology Maturity Model (TMM)
About the Authors
Dear Educator (Top of Page)
Technology is transforming the way we learn, work, socialize, and communicate. The full integration of technology into our schools is required for students to be successful as learners today, and as future workers in an information-driven economy tomorrow.
Instructional technology provides unprecedented opportunities for students, and has become a national priority for educators, legislators, and community members. This priority has been recognized with landmark funding initiatives and legislation such as E-rate programs and Technology Challenge initiatives. Experts predict that overall spending on education technology will increase by as much as 38% over the next few years. These projected increases have created a need for knowledge-based leadership as well as tools for the planning and deployment of instructional and administrative technologies.
The following themes continue to influence the work of educators fundamentally shaping the future of applied-technology education reform:
- Applied Educational Technology
TechBuilder 3.0 addresses the need for comprehensive assessment, standard-based application and performance based Technology Use Planning.
Compaq Computer is committed to not only providing the best technology solutions in the industry, but also to providing leadership and direction for educators. TechBuilder 3.0 is representative of Compaq's commitment to leadership by providing tools that will facilitate effective planning, implementation, and assessment of educational technologies.
TechBuilder 3.0 presents practical and proven strategies for planning technology to improve student learning and impact systemic organizational change. This guide has deliberately focused upon tools for school leaders that will facilitate effective technology planning and implementation. In particular, the strategies presented in the guide will be valuable for:
- District and School Technology Coordinators
- Technology decision-makers and planners
- School Administrators
- District Administrators
- Board Members
The purpose of the TechBuilder 3.0 is to provide:
- A means of building assessment strategies into the technology planning process
- A proven process for planning and implementation
- Practical tools for technology planning and assessment
- Resources for schools and districts to assess their capabilities, and the impact of technology upon student learning
- Concrete help for schools to integrate technology into the classroom
Techbuilder (Top of Page)
TechBuilder 3.0 is a web-based technology-planning tool based on the Technology Maturity Model© (TMM).
Developed by the education experts at EDmin.com and used by more than 2,500 districts nationwide, TechBuilder can be used to assess:
- What technology is currently being used in a single classroom, school or district
- How technology is currently integrated into the curriculum and administrative activities
- How technology can be better utilized to improve student learning
TechBuilder 3.0 is available at http://compaq.edmin.com
The Growing Need for Assessment and Accountability (Top of Page)
Assessing educational institutions has become a performance-based monitoring method. Many states support legislation that require the evaluation of student performance and program effectiveness in order to allocate and continue resource funding. Significant numbers of education administrators believe performance-based assessment will maintain or increase in priority over the next three to five years. Many experts believe that educational institutions will become increasingly accountable for student's performance as schools are closely evaluated to determine the level of funding.
TechBuilder 3.0 will capture the assessment results of an individual school, school district, educational projects/ programs, and student assessments. These results can help your organization identify its strengths and where additional effort and resources should be focused. In addition, the entire school community -- including parents, school boards, business leaders, and legislators -- will benefit from the ability to review a comprehensive plan outlining your organization's committed to improvement and growth.
The Five Challenges (Top of Page)
In examining schools and districts across the country, we have identified the following challenges:
- Vision: Developing a vision for powerful teaching and learning including the use of technology
- Planning and Assessment: Establishing a planning process for technology use and appropriate assessment processes to establish its effectiveness
- Support: Implementing support systems necessary for success, including staff development and technical support
- Literacy: Understanding the technology
- Communication: Collaborating and communicating so that all the stakeholders are engaged, involved, and valued
The TMM addresses these challenges through the use of the Improvement Cycle and the Maturity Indicators and the TMM Tools. These components of the TMM are derived from the requirement to define both processes and products in a manner that reflects "best practice." The most powerful aspect of the TMM is the way it promotes ongoing assessment. The "Maturity" of the organization and its ability to realize the full potential of technology is determined by its assessment practices. The longitudinal assessment efforts described in the TMM tend to be more accurate than one-time studies. They also have the capacity to document long-term changes and patterns of growth.
Introduction (Top of Page)
The Technology Maturity Model (TMM) is a powerful methodology for developing a comprehensive approach to using technology to improve your institution and the education it affords your students. "Maturity" refers to an organization´s ability to systemically realize the potential of instructional technology. The benefits of the use of technology must be planned and implemented throughout the entire organization, not just a few points of excellence. The TMM was developed in response to the need for appropriate accountability as increased financial resources are allocated to technology. The TMM supports expert views of assessment which call for a multidimensional approach, using both qualitative and quantitative measures for program and institutional effectiveness. It also embraces a culture of longitudinal data collection for systemic improvement.
Experience has shown that many educational institutions have underestimated the impact of technology, and the need for comprehensive processes for planning and evaluation. In addition, the established culture of many institutions is to produce documents and programs, not measure results. The lack of commitment to action based on assessment undermines a culture of thoughtful planning and measured improvement.
An Introduction to the Technology Maturity Model (TMM) (Top of Page)
Assessment is one of the key components of the Technology Maturity Model. The TMM distinguishes itself from other planning models by establishing benchmarks that evaluate resource availability and effective resource allocation. Additional components include:
- Improvement Cycle: A means of implementing refinement and organizational improvement
- Maturity Indicators: Benchmarks for measuring results
- TMM Tools: Processes instruments and applications that improve effectiveness
Note: For further description of the key components see below.
Improvement Cycle (Top of Page)
The Improvement Cycle component of the TMM describes processes that can be used to support planning for most organization activities. The result of the Improvement Cycle is a comprehensive action plan, with an established process of on-going planning, implementation, and assessment cycles.
The action plan is a result of a series of assessment activities that benchmark your organization's capacity for successful implementation. The Improvement Cycle and the results of the activities suggested, provide the conceptual framework for the technology implementation efforts in your organization.
Maturity Indicators (Top of Page)
The Maturity Indicators provide a means of assessing the current levels of use of technology within an organization. The Maturity Model Benchmarks, for instance, assesses resources availability and effective resource allocation. Using the TMM Maturity Indicators you will be able to clearly focus your improvement efforts.
TMM Tools (Top of Page)
The TMM Tools will help you through the Improvement Cycle. The TMM Tools will both facilitate the planning process and provide critical information about your organization. The tools presented in TechBuilder 3.0 are designed to be flexible so that you can choose to use any or all of the tools, and you can customize them according to the needs of your organization
The Technology Maturity Model defines specific steps that assure your institution's goals in the technology plan are translated into specific projects with measured results. A single school, district or other comprehensive educational agency can use the TMM. The TMM has been refined over a ten-year period and used by over a thousand educational institutions. The TMM is a significant resource whether you are just beginning the planning process or if you have a plan in place and are seeking processes and tools to refine the work that you have already completed. These benchmarking processes, tools, and results will provide a powerful strategy for your institution. Further, the use of the TMM will support efforts to provide effective leadership for your school or district.
The chart below describes the tools provided in TechBuilder based upon the Technology Maturity Model. It is our hope that these tools will facilitate the improvement process and technology efforts in your organization.
Technology Plan Analysis Rubric
Used to evaluate your current technology use plan to identify priorities for the planning process.
Staff Needs Assessment Survey.
Used to evaluate staff technology skill, training, and opinions on best uses of technology.
Hardware Resource Survey
Used to develop a comprehensive picture of the current technology assets within the district.
PlanBuilder gives step-by-step instructions on how to build a comprehensive technology use plan.
About the Authors (Top of Page)
The Technology Use Planning Guide was authored and produced for Compaq Computer Corporation by EDmin.com of San Diego, California. EDmin.com is an educational consulting, systems integration, and Internet software development firm committed to the effective implementation of technology in education, government, and health care. The content of this toolkit is based on The Technology Maturity Model developed by Peter H.R. Sibley and Dr. Chip Kimball.
The Technology Maturity Model (TMM) was developed over a ten-year period drawing from educational research, consulting with districts nation-wide, and the expansive experience of the development team. Peter H.R. Sibley currently serves as the Chief Executive Officer of EDmin while Dr. Kimball is serving as an Assistant Superintendent for Technology in a large progressive school district.
For questions about technology planning services or the Technology Maturity Model, EDmin can be contacted at (619) 296-8090 or firstname.lastname@example.org.
For questions about Compaq solutions that will meet your organizationís needs, contact your local Compaq Sales representative or call 1-800-88-TEACH to locate the representative nearest you.
Organizational Phase (Top of Page)
Each Improvement Cycle begins with the Organizational Phase. During this phase the Technology Committee is established and given formal recognition. The committee charter, which defines the work of the committee, is developed and approved. The steps to complete the Organizational Phase are:
1. Determine "to whom" the committee will report.
The work of the committee will require operational authority. The plan itself and its recommendations must be presented for review, prioritization, and approval. The reporting relationship identifies the primary audience for the plan.
2. Determine the scope of the technology planning effort.
Scope refers to the organizational units to be included in the work. Commonly, the whole school or district is included in the technology use planning process, but not always. Sometimes, for example, instructional and administrative needs are addressed separately. It is best these issues be clarified and special instructions specified.
3. Determine the committee membership.
Broad stakeholder representation is valuable on the technology committee. Certified and classified representatives as well as community members and students help develop a plan that includes all perspectives. Broad stakeholder representation contributes to broad based support essential for success.
4. Determine the budget for developing the plan.
Resources to develop the plan will be required. The budget should include everything from copying costs to travel expenses to visit effective users of technology.
5. Determine the timeline for the plan to be completed.
Ideally, technology use plans are revised annually and become an established part of the organizations broader planning processes. Still, deadlines may require a less that ideal timeline. The committee must be informed of the required milestones for completion.
6. Build committee expertise.
The committee will find it does not typically posses all the expertise it requires to complete its work. The rapid changes in technology itself contribute to this dilemma, as well as the relative scarcity of information on the effective use of technology for improving student learning. Find ways to build expertise within the committee.
Assessment Phase (Top of Page)
The Technology Use Plan (TUP) is developed or revised in the Assessment Phase with two major benefits:
First, when a Technology Use Plan is created, components of enduring value are developed such as the vision, mission, goals and objectives. These components should align with those found in other organizational planning efforts such as school improvement efforts or strategic planning. These elements tend to be overarching in nature. They are strengthened when linked to strong assessments of organizational needs and to descriptions of projects that address those needs is crucial to successful technology use planning.
Second, when developing the TUP describing technology use projects at a summary level saves the technology committee's time and energy. Many technology plans err by including project details that are not needed until a project is actually approved and funded. The emphasis in the project description is on the benefit it provides to the organization, not on detail planning. The project descriptions are more enduring since they do not describe products that become quickly obsolete. Vendor products used in implementing technology use projects are identified only after the projects are approved and funding allocated. This approach can save a significant amount of time.
Assessment criteria and survey instruments are selected which will typically be reused in future iterations of the Improvement Cycle. Plan to collect data for assessment from the beginning. Areas of assessment should include: integration of technology into all curriculum content areas, teacher and student needs assessment, staff development, resource availability and capability, support and maintenance of technology, and survey of other stakeholders.
The technology use plan is most importantly a communication document. It serves to inform the governing body and the stakeholders how technology can be used to meet the stated goals and objectives and to report progress in attaining the goals. Technology Use Plans often refer to other documents or use appendixes to keep the message from being overwhelmed with detail.
The ten steps to complete the Assessment Phase are:
1. Continue to Build committee expertise.
Plan to engage in activities that will build committee expertise at every meeting. This can be as modest as sharing technology use plans from other organizations to vendor presentations to onsite visits of high capability organizations. Avoid reinventing solutions and concentrate on improving implementations that have value.
2. Develop a powerful vision for teaching and learning.
Powerful visions for teaching and learning thrive on trust and creativity. Resources are available, from books and videotapes, to workshops and retreats to assist in developing your vision. Use other organizational resources such as grants and planning documents to feed your efforts. If you have a powerful vision described already, adopt it.
3. Develop a model for assessment.
At the heart of successful planning is objective, reliable, accurate measurement. The assessment model must include a means of evaluating process and products necessary to satisfy the TUP's vision, mission, goals and objectives. It is not always possible to measure the goals and objectives directly. Process and product measurements must be developed, which when correlated, predict success. Process measurements include budget, timelines and effort. Product measurements include test scores, availability of resources, and survey results. Some measurements qualify as both a product and a process measurement. The assessment model describes the functional relationship between the measurements and the desired outcomes. An example of such a relationship is a goal of making teachers effective users of technology. A survey that records the level of teacher use of technology, self-assessed competencies, instructional value of various technologies, and areas, which in which staff development at certain intervals, is needed.
The Needs Assessment section in TechBuilder draws a relationship between these measurements and making teachers effective users of technology. Higher scores in each area are predictive of teachers being more effective users of technology.
4. Develop/select measurement instruments to support the assessment model.
The surveys and measurement instruments collect data for analysis. One instrument may collect a variety of measurements or just one type of measurement. Every goal must have a relationship to a measurement and each measurement must have data. The first time these measurements are taken become internal benchmarks for future comparison. Some measurements are taken and reported by organizations regionally or nationally. Comparison with other organizations allows external benchmarking and immediate comparisons.
5. Collect assessment data.
Three essential measurement instruments are included for your use: the Technology Maturity Model Survey, the Needs Assessment Survey, and the Infrastructure Survey. These instruments provide a foundation for assessment that collects a wide range of valuable information. Each instrument is also available via the web for comparison with other organizations.
6. Summarize, analyze, and evaluate assessment data.
Use the relationships developed in the assessment model to analyze your organizations initial state to the desired state. Use the data to inform the committee and frame the deliberations. Select representative data for use in the assessment and finding section of the TUP. Do not feel compelled to put every possible graph in the main body of the TUP. This is a perfect use of appendixes for the reader who may wish to explore the data in detail.
7. Develop recommendations.
Recommendations will generally flow from the assessment data. Sometimes the recommendations are obvious prior to the collection of data. Even so, an essential element of the Improvement Cycle is that the data is used to inform and substantiate recommendations. Recommendations generally fall into three areas: policy recommendations, which require Board approval, procedure recommendations, which require administrative approval, and resource recommendations, which may require both Board and administrative approval.
8. Group recommendations.
Recommendations are the conceptual links that tie desired outcomes to allocation of effort and resources described in the project plan. The Technology Maturity Model groups organizational criteria around five organizational filters: administration, curricular, support, communication and innovation. These groups afford the committee the opportunity to assess the impact of recommendations. Some plans have focused recommendations on one area and neglected others. Using the Benefits Impact Table, your recommendations may be assessed in light of the audiences impacted.
9. Develop technology use projects.
Technology use projects translate recommendations into the application of resources to satisfy goals and objectives. It is the technology use projects that institute change.
10. Submit the technology use plan for approval.
Stakeholders review the Technology Use Plan and revisions are made. After a final review, the plan is submitted for approval, prioritization, and funding.
Formulation Phase (Top of Page)
Technology use projects are translated into a project plan in the Formulation Phase. It is important to draw a distinction between a technology use project and an infrastructure project.
Technology use projects focus on the application of technology providing an overall organizational benefit. Using technology to improve student learning is a "use" of technology. Providing students with electronic research resources is a technology use project. On the other hand, building a local area network is an infrastructure project. Infrastructure projects build the capacity required to support technology use projects.
Educational organizations presently find that nearly every technology use project requires an infrastructure project. Infrastructure projects include wiring buildings, installing local and wide area networks, acquiring desktop computers and purchasing software. Infrastructure projects build capacity to do things with technology. Technology use projects employ the capacity created by infrastructure projects to attain the organization's vision, fulfill the mission, and satisfy goals and objectives.
Historically, educators who were primarily interested in technology use and not in the technology itself felt alienated by the technical complexity associated with building technical capacity. Instructional leaders must guide the development of the technology use plans. They must make sure that the capacity building associated with the technology use project serves the organization's goals. Yet, instructional leaders should not be drawn into inappropriate levels of technical complexity. Technology use projects will eventually require less capacity building. In the future existing technical capacity will be augmented and re-purposed from project to project with relative ease and requiring fewer resources to support a typical technology use project.
Detail project plans are developed in the Formulation Phase for projects with approved funding. Only approved projects have detailed plans developed. This saves time by not needing to develop "detail plans" for projects that are not authorized. If two or more projects are approved, each is typically managed separately. Combining projects can add complexity, unless the projects are small or you have a clear understanding of the project.
The approved projects are usually phased-in over multiple funding periods. If this is a new project, phases will need to be defined. The first phase cannot use more resources than was approved. The challenge of defining phases is to get the most benefit with resources allocated. If you buy all your equipment in phase one, you may not have resources for training, software, network installation, communication costs and other items that make the project work. We have all seen the waste and felt the frustration associated with poorly phased projects. Future phases may not have firm budget commitments so they are estimates of work to be completed and should not require detail planning.
Individuals with specific knowledge of the project often undertake the actual development of the detail plans. The Technology Committee may have an oversight role based on its charter, or act in an advisory capacity. Detail project plans include project scope, budget, timeline, tasking, specifications of work, and change control. This may sound complicated if you are not familiar with project planning. In fact, small and medium sized projects may take only a few hours to develop into a detail plan sufficient to manage the project effectively.
Having an experienced project manager can save time and increase the effectiveness of your efforts. Large projects that require significant capacity building often require the work to be competitively bid. The Technology Committee may contract with a consultant to assist in managing the bid process and provide oversight of the work being contracted as well.
Established projects, which have received approval for additional funding, usually have a planning structure in place. Even so, review of project plans and refinement will assure projects do not fail as a result of poor planning. The summary project descriptions developed in the Assessment Phase provide an initial structure. Project definition and detail is added in the Formulation Phase to assure projects can be successfully managed and implemented.
The steps to complete the Formulation Phase are:
1. Review technology use project summary(s) provided in the TUP.
Each approved project should be reviewed to determine if the project requires significant capacity building. Special technical expertise will be needed for projects requiring wiring and infrastructure; local and wide area networks; new or sophisticated hardware and software; or other technical capacity. Just as importantly, technology use projects, which impact curricular areas will need, content experts. Identifying early on expertise needed to develop the project will aid immeasurably in the projects eventual success.
2. Assign project responsibility to qualified individuals.
The project manager is selected. If necessary, other project team members are recruited. The project manager should be given access to specific expertise if required. This may range from curriculum consultants to network engineers depending on project needs. The project manager reports to the committee at agreed upon intervals.
3. Conduct a site survey.
Technology use projects require consideration of electrical systems, air conditioning, security, data communication, furniture and even space. Survey sites where technological resources are to be located for adequacy.
4. Develop detail project plans.
The project team develops the detailed project plans. This may include an expanded project description; revised scope to reflect phasing of project over multiple budget years; work specifications, timeline, budget, staffing plan, training plan, and support plan.
5. Submit detail project plans for approval.
Detail planning usually uncovers areas needing revision. Minor changes may include moving budget allocations within the approved budget. More significant changes include changes in the project scope or amount budgeted. If large variances are anticipated, the project may need to be reworked.
6. Move approved plans to implementation phase.
Each project plan, once approved, moves to implementation.
Implementation Phase (Top of Page)
Technology use projects are implemented based on the project plan developed in the Formulation Phase. Resources are allocated equipment purchased, infrastructure capabilities enhanced, training provided and support systems augmented. At the conclusion of the Implementation Phase the technology use project is operational.
The steps to complete the Implementation Phase are:
1. Identify lines of communication.
Technology use projects have the ability to change organizations. Resistance to change thrives on a lack of communication. Identify the stakeholders in the project and define lines of communication to improve acceptance of the project and identify problems early.
2. Purchase process reviewed.
The purchase process for the technology use project is reviewed to assure compliance with local, state and external funding requirements. Procurements associated with technology use projects may need to be bid or notices served to assure a fair process. It is the project manager's responsibility to make sure of compliance with all regulations.
3. Execute the plan
Seasoned project managers offer the timeless advice of "Plan the work and work the plan." The planning effort and products developed in the Formulation Phase are used to guide the implementation. Expect that changes to the plan will occur and modify the plan, but do not ignore the plan.
4. Award contracts.
Contracts large or small are usually awarded as part of the technology use project. Select your vendors carefully. Many vendors have a history of commitment to educational institutions that goes beyond just a "dollar and cents" evaluation. The best choice for a vendor may have invested in sales and support tailored to your needs. As they work with other educational organizations, they learn what works and will share that knowledge with you. This may be your only project using technology, but an experienced vendor is working on similar projects everyday. The experience of the "right" vendor is a great resource.
5. Verify completion of work.
Use the detail descriptions of products to verify you have what you purchased. It is not uncommon to find elements of the project missing, such as: the current version of the software; documentation; or even entire computers. Once you have completed your purchases, check that training, installation and non-tangible items were satisfactory. Make a list of goods and services not received or that were unsatisfactory. Bring all unsatisfactory work or products to the vendor's attention as soon as possible and typically within 30 days of receipt of the good or service.
6. Verify project is operational.
Your project began with a summary of features and capabilities required to deliver the promised benefits. The features and capabilities evolved into detail descriptions of products to be purchased and installed and services provided. It is important to check that the system is operational. Compare the features and capabilities promised with those delivered.
7. Report on phase status.
Most projects are implemented in phases over multiple budget years. Once the current project phase is operational, a phase is concluded. In new projects this is the end of the first phase. For existing projects this might be the conclusion of phase two or three. Document the operational status of the project. If the project is experiencing difficulties, even if they cannot be resolved immediately, document the problems. If the project seems to be operating at or above expectations, document that as well. Include a legacy report that shares what you would suggest doing different in a future phase or improvements that may be necessary.
8. Final sign-off
The project is operational. Now it is time to shift from making the project operational to making the project fulfill the intended organizational goals. Organizational support systems become responsible for the operation of the project at the conclusion of the Implementation Phase.