Teacher and Leader Preparation Implementation Committee Meeting via Webinar
February 1, 2012

TLPIC Timeline
•	Fall 2011 – Spring 2012
•	Recommend performance targets for three levels of continued approval and one for denial
•	Spring 2012 – Summer 2012
•	Continue standards revision recommendations including Uniform Core Curriculum and site visit protocols
•	Summer 2012
•	Finalize draft recommended continued approval standards and performance targets
•	Provide recommendations/information to FDOE Program Approval Office for Pilot Site Visit implementation

Steps to Reach Spring 2012 Deadline

February 1, 2012
•	Review Data Elements Table and Recommendations from AIR

•	Finalize Data Elements List and Business Rules/Decision Items

•	Homework: Weight Data Elements List

Review Data Elements Table and Recommendations from AIR
Data Elements Handout 

Review Data Elements Table for: 		

•	Placement Rate Data				
•	Retention Data				
•	Critical Teacher Shortage Area Data		
•	Employer Satisfaction Data

Data Element – Placement Rate Data
Metric Options  -
1. The percentage of completers who become employed in an instructional position in a Florida public school district their first year following program completion is included in the accountability system.  
2. The percentage of completers who become employed in an instructional position in a Florida public school district their first or second year following program completion is included in the accountability system.
Decision Point Questions  - 
How many cohorts of completers should be included?  Could be anywhere from 1 to 5.  Use of a minimum of 3 cohorts of completers would allow smaller institutions, districts, or programs an opportunity to reach the N of 10 threshold. 
Should this element be considered at the institution-district level or at the program level? If at least 3 cohorts of completers are used, data should be sufficient to allow incorporation of the element at the program level.

Data Element – Retention Data
Metric Options – 
1. Based upon the decision made with placement rate, the average length of stay in an instructional position in a Florida public school district across five years of employment is included in the accountability system. 
2. Based upon the decision made with the placement rate, the percentage of completers continuously employed in an instructional position in a Florida public school district at the third year and fifth year marks is included in the accountability system.
Decision Point Questions – 
How many cohorts of completers should be included? Could be anywhere from 1 to 5.  Use of a minimum of 3 cohorts of completers would allow smaller institutions, districts, or programs an opportunity to reach the N of 10 threshold. Should this element be considered at the institution-district level or at the program level?  If at least 3 cohorts of completers are used, data should be sufficient to allow incorporation of the element at the program level.

Data Element – Critical Teacher Shortage Area Data 
Metric Options –
1. The percentage of completers who become employed in a critical teacher shortage area instructional position in a Florida public school district their first year following program completion is included in the accountability system. 
2. The percentage of completers who become employed in a critical teacher shortage area instructional position in a Florida public school district their first or second year following program completion is included in the accountability system.
Decision Point Questions- 
How many cohorts of completers should be included? Could be anywhere from 1 to 5. Use of a minimum of 3 cohorts of completers would allow smaller institutions, districts, or programs an opportunity to reach the N of 10 threshold. Should this element be considered at the institution-district level or at the program level?  Even with inclusion of multiple cohorts many programs would likely not meet the N of 10 threshold if this element is included at the program level. 

Data Element – Employer Satisfaction Data  
Metric Options – Committee has decided to use the percentage of completers that employers indicate meet their criteria to be rehired.
Decision Point Questions – 
How many cohorts of completers should be included? Could be anywhere from 1 to 5. Due to low survey response rate, use of a minimum of 3 cohorts and possibly up to 5 cohorts of completers would allow smaller institutions, districts, or programs an opportunity to reach the N of 10 threshold. 
Should this element be considered at the institution-district level or at the program level? Due to low survey response rates, even with inclusion of multiple completer cohorts many programs would likely not meet the N of 10 threshold if this element is included at the program level.  

Data Elements Handout 

Review Data Elements Table and AIR Recommendations for:
			
•	Value-Added Model Data				
•	Student Performance by Subgroups 	
•	Teacher Evaluation System Results		
	
Data Element – Value-Added Model Data
Metric Options - Committee has decided to use VAM data within an accountability system with certain considerations:
•	In-program/In-field.  Only include VAM data on completers who were trained in-program and teaching in-field.  If VAM data is not aligned with area of certification, it should not be actionable.
•	Sample Size.  As a default, a minimum sample size of 10 completers with VAM data should be used.
•	Completer cohorts to include.  Include VAM data for completers employed in an instructional position in a Florida public school district their first or second year following program completion.  

Metric options to consider regarding VAM data:
•	Average VAM scores of completers from a particular institution/district
•	One-year snapshots or an aggregation across years?  How many years?
•	Use of standard error and confidence levels

Options to consider regarding the issue of in-program/in-field completers:
•	Aggregate all VAM data (regardless of in-program/in-field) when evaluating institutions overall; OR
•	Aggregate all in-program/in-field VAM data when evaluating institutions overall (not including out-of-program/out-of-field results); OR
•	Limit VAM data to in-program/in-field completers when evaluating specific programs.

If limiting VAM data under consideration to only in-program/in-field completers, what options do we consider for EPIs and DACPs?

 Committee has decided to use VAM data to compare programs based on the standard of “effective” teachers. That is, state approved programs should produce “effective” teachers upon completion. At this time, the state has not determined a standard to define “effective” in terms of performance on statewide assessments. A standard for use in district teacher evaluations plans beginning in 2012-13 is expected to be set by August 2012.   

Metric options to consider:
•	Prior to the establishment of a statewide standard for “effective” choose an alternate standard for piloting purposes.
•	Examples include, the statewide average, some degree above average (e.g., 10% better than average).

Decision Point Questions – 
How should VAM data be incorporated in the evaluation of teacher preparation programs?
•	AIR recommends yes VAM data should be included in the evaluation of teacher preparation programs, though a number of considerations must be taken into account (delineated below).
•	AIR also states that strong consideration should be given to the weighting of this information in the decision due to the limited number of completers with VAM scores; program approval should not be based solely on the VAM scores of completers.
When using VAM data to evaluate teacher preparation programs, AIR noted several technical considerations that should be taken into account.  Notably, (1) standard error; (2) level of statistical confidence; and (3) sample size.

Should standard error be taken into account in evaluating institutions and districts?
•	AIR notes that standard error should be taken into consideration when making determinations about program approval and accountability.
•	The standard error provides an indication of the range of VAM scores with which the observed score is consistent.

Should and, if so, how should thresholds of statistical confidence be used in evaluating institutions and districts?
•	AIR recommends that Florida use at least a 68 percent level of confidence (i.e., one standard error) and preferably a 90 percent level of confidence in comparing the performance of institutions and districts.
•	In determining the level of confidence, consideration must be given to the ability to distinguish performance (more likely when using lower levels of confidence) and the risk of misclassifying institutions and districts (less likely when using higher levels of confidence).

In order for the VAM scores to be considered valid, what is the minimum number of completers that should be used?
•	AIR notes that the TLPIC has recommended that at least 10 completers with VAM scores be included.
•	AIR recommends that in addition to a minimum number, the proportion of the total number completers who have VAM scores should be considered so that the outcome is representative of the institution or district.

How many cohorts of completers should be included?  Could be anywhere from 1 to 5.
•	AIR recommends that data should be combined across years, if possible.  
•	When the data are combined over three years for program completers, the overall precision of the information is improved, making it easier to distinguish the performance of institutions and districts.

Should the completers used in the analysis be limited to those one-year out of program completion?
•	AIR recommends that only completers one-year out should be included in the approval and accountability system to prevent the introduction of experience and other time-related variables.
•	Additionally, only a single year of growth data for completers one-year out should be used for a particular cohort of completers, even if later years of data are available.

Data Element – Student Performance by Subgroups
Metric Options – 
Committee indicated a desire to include student performance by subgroup as an enhancement to an accountability system for teacher preparation programs.  

Metric options to consider:
•	Percent of students within each subgroup meeting/exceeding expectations
•	Comparison of subgroup performance to a standard
•	Number or percentage of subgroups that exceed established standard as a determinant for program recognition.

Decision Point Questions – 
If student subgroup performance is to be included, AIR notes the following areas in need of careful consideration:
•	The complexity of using student subgroups
•	Determination of which student subgroups to be used – in a fair and consistent way with all institutions and districts
•	Procedures for cases where institutions or districts may have completers who did not serve particular subgroups or small numbers in particular subgroups.
•	The basis on which recognition is to be awarded, and the overall impact of this feature on overall program evaluation. 

Data Element – Teacher Evaluation System Results
Metric Options – 
Committee has expressed a desire to include the new teacher evaluation results for all completers in the evaluation of program effectiveness.
Metric options to consider:
•	Percentage of completers evaluated as highly effective, effective, needs improvement/developing, or unsatisfactory
•	Focus on specific elements of the evaluation system (e.g., instructional practice and/or student performance)
•	Evaluate performances in the context of the districts the completers are employed?  The state?  Other standards? 

Decision Point Questions - When overall teacher evaluation results are available (earliest by Fall of 2012), decisions will need to be reached on how to incorporate teacher evaluation results (e.g., the overall ratings, elements of the instructional practice and performance of students portions of the evaluations) into an accountability system for teacher preparation programs.

Next Steps
•	Homework: Complete survey weighting the data elements by assigning a percentage to each element (should total 100%). http://www.survey.coe.fsu.edu/TakeSurvey.aspx? 
					Due February 6th 

•	DOE will compile results from your survey and combine them with your decisions today to run appropriate data (as available) and report various weighting options back to the committee during the March 8th meeting.

Steps to Reach Spring 2012 Deadline
•	March 8, 2012
•	DOE data presentation and recommendations using results from committee input on weighting of data elements.
•	Committee discussion on:
o	 cut points
o	 performance categories
o	 bonus points

•	May 9-10, 2012 
•	Finalize recommendations for performance targets (3 for approval, 1 for denial)