RISE Evaluation of Seminars Revised

The student development portion is supported by exposure to a inquiry-based curriculum,research experiences, presentations of research results, participation in seminar series andspecial

UNIVERSITY OF THE VIRGIN ISLANDS RESEARCH INITIATIVE FOR

SCIENCE ENHANCEMENT (RISE) PROGRAM: MID YEAR EVALUATION REPORT AUGUST 2001-JANUARY 2002

Background

The RISE program is one of several collaborative training/research programs in the Sciences Division of the University of the Virgin Islands These programs include: (1) The Minority Access to Research Careers ( MARC) program funded by NIH, a

faculty/student research program conducted, in part, in mainland university labs, (2) a NASA scholarship program, and a National Science Foundation (NSF) grant designed to promote excellence in science, mathematics , engineering and technology These

programs are under the umbrella of Emerging Caribbean Scientists program and

commonly share the goal of increasing the number of UVI students who are skilled, motivated and confident about their preparation for biomedical and other scientific careers

Overview of the RISE Program

The goal of the RISE Program is to overcome the geographic and intellectual isolation of the University of the Virgin Islands and increase the number of graduates that are skilled, motivated, and confident about their preparation to enter graduate studies in the sciences This is accomplished through a three-step process involving students, faculty and The University of the Virgin Islands Collaboration is structured between UVI faculty in different biomedical disciplines, students, the university, and mainland scientists in approved laboratories

The student development portion is supported by exposure to a inquiry-based curriculum,research experiences, presentations of research results, participation in seminar series andspecial workshops, clubs, study groups and travel to mainland universities Participation

in research programs on the mainland provides opportunities to interact with role models and broaden experiences in a student’s chosen field

The faculty development portion provides faculty the opportunity to initiate

collaborations with outstanding research scientists and travel to mainland laboratories with the goal of enhancing their professional careers and productivity

The institutional development portion provides renovated space, capital equipment, materials, staffing, upper level course redesign to modernize molecular biology training,and the revamping of general chemistry

Description of RISE Participants and Mentors

1 RISE Participants 2001-2002

There are ten students supported by the RISE program in the 2001-2002 academic year Eight are female and two are male Four are sophomores, four are juniors and two are

seniors All participants have a GPA of 2.5 or better, are US citizens and are classified as

a minority student All students selected for the RISE program have demonstrated, by participation in other research based activities and written documentation, an interest in a career in biomedical research and motivation to obtain a Ph.D In addition to the ten students who are the cohort for the 2001-2002 school year, three additional students were designated RISE students during the summer term, and received support for mainland research projects

Mentors

Mentors for the RISE Fellows consist of two groups : There are five UVI based mentors

at the University of the Virgin Islands and three at mainland universities throughout the United States The criteria for being a mentor are being an experienced researcher associated with a funded research laboratory.( NSF, NIH, etc.) and a willingness to mentor undergraduate students

1 Students will be provided the support to excel in gatekeeper courses (general

chemistry, general biology, calculus, and physics)

2 Retention of RISE freshmen students in biology, marine biology and chemistry

programs will exceed 60% in school year 2002-2003

3 Student will have greater interest in and awareness of diversity of biomedical research after attendance at seminars and scientific meetings as measured by survey responses

4 Students will become active scientific researchers as demonstrated by presentations atscience seminars, scientific meetings, and reports from mentors At least 2 RISE

students will co-author publications in refereed journals

5 Students will be knowledgeable in their field as demonstrated by course grades, mentorand faculty evaluations

6 Student confidence will increase as demonstrated by performance in course work, application to graduate programs, scientific presentation, and co-authorship of articles

7 Faculty in the Division of Science and Math will develop significant collaborative relationships which will lead to productive research careers as measured by progress reports from summer research experience, submitted manuscripts and grant proposals.8a.Modernized molecular biology instruction as demonstrated by refurbished lab,

modular course development and implementation

8b.Restructuring of the General Chemistry curriculum

Evaluation Plan of the RISE program

Introduction

The goal of the RISE Evaluation Plan is to measure the progress in achieving the stated goals and objectives of UVI RISE by assessing whether stated objectives are being met, activities implemented, and the efficacy of methods, materials, curriculum, and design being utilized to achieve them The evaluation is a collaborative effort among MBRS-RISE staff, UVI office of Institutional Research, Program Directors of other MBRS projects , and the outside evaluators, Lce Consulting Evaluation is formative as well and summative in order to provide timely feedback to stakeholders which provides the

opportunity to improve the project as it develops The ongoing evaluation process gathers data during the course of the project The MBRS-RISE staff collects, tabulates, and forwards to the external evaluators a variety of data gathered on RISE students withinthe University’s program The External evaluators analysis and interpret this data in addition to designing instruments to gather data, conducting observations, interviews, focus groups, record reviews, research and providing a written report

Evaluation Plan

Table 1 ( appendix 1) presents the evaluation questions that framed the evaluation design, methods of data collection, and responsibility allocations Figure A summarizes the research methods and data collection used to complete the present evaluation

Overview of Evaluation Activity August 2001- February 2002

RISE evaluation activities consisted of observations of Poster Presentations ( July 30 and September 22), design of student and mentor survey forms (see appendix 2),

administration of student and mentor survey forms, faculty/mentor interviews,

FIGURE A

RISE: EVALUATION METHODS AND DATA COLLECTION

O BSERVATION AND EVAL

A-TION: POSTER PRESENTATIONS

July 30 September 22

STUDENT: RISE PROGRAM SURVEY FORMS

- Summer Research

- Presentations

LCE EDUCATIONAL CONSULTING

EVALUATION ACTIVITIES AUGUST 2001 - JANUARY 2002

participation/ presentations at the joint MBRS Marc and RISE advisory council meeting, meetings with RISE project Director, conducting focus groups, document reviews, data collection and analysis, and reporting

Psychology The three poster presentations were observed, photographed and RISE presenter interviewed The evaluator, Dr Creque, gathered data on confidence level, opinions on summer mentoring experience and plans for graduate school Each

participant was observed for approximately 25 minutes

Confidence level was defined as the ability to:

 -answer all questions posed by visitors and evaluator

 -explain poster exhibit in detail

 -speak knowledgeably about content

 -make eye contact with visitors and observer

 -remain relaxed (smile, speak slowly and distinctly, etc)

Each criterion had a value of 20 Using these criteria , during the period of observation allthree students met the five criteria of exhibiting confidence (minimum scored: 80) Summer Mentoring Experience:

RISE participants were asked to describe their summer mentoring experience All responses were positive Comments were as follows:

Got to see what I was doing Wonderful to have one-on-one.

I had an opportunity to participate in almost everything The experience more intense than two weeks in a regular class.

The research encouraged me to go on to graduate school (I see) what I am moving toward, I can only achieve at the Masters level.

Helped me to focus on what to study in grad school

Plans for Graduate school

All three students interviewed plan to go to graduate school, however, their discussion and comments center on the Master’s level Only one student, when asked, responded thecontinuing on to a Ph.D was “possible.”

Development and Administration of RISE Student and Mentor Evaluation Forms

RISE Students

Survey forms were designed to collect baseline data on RISE students assessment of

their likelihood of entering a Ph.D program, the quality of counseling and support , level

of proficiency in their field and awareness of the biomedical or other research areas after

their summer RISE experience These questions relate to the overall goal of the RISE

program and specific objectives, 2,3, 4, and 6

RISE Mentors

Survey forms for RISE mentors paralleled question in RISE Student Self-report survey

instrument Mentors were asked to assess how likely it was for their mentee to enter

Graph 1: Student/Mentor Comparason:

Preparation For GRE

Passing Oral Exam

Making paper presentation at conference

CO-authoring articles

Students Mentors

awareness of biomedical and other research fields Mentors were also asked to assess their ability as a mentor.

Direct contact was made with RISE Students and Mentors to explain the survey and respond to concerns Mentors located on the mainland were telephoned Only one mentor was not spoken to directly and contact was made through Email

Results: RISE Program for 2001

There were seven students and five mentors that responded to the program evaluations Six out of seven of the students said that they intended to pursue a Ph.D program Of those expressing a desire to continue, 5 students said that they were planning to wait at least a year before going on, and one student said that they were planning to go in the next 10-12 months The person who said that they didn't want to go on said that they wanted to go into medicine rather than get a Ph.D Some of the universities students are considering are Virginia Commonwealth, Howard, Georgia State, Florida State, Georgia Institute of Technology, Stanford, Boston and Ross

Both students and advisors tended to rate nearly all areas of the quality of academic advising high Graph 1 compares student and advisor responses There were only two areas where there appears to be a significant difference between the means of the student and mentor responses (p <.05) These are in Course Selection and in Encouragement to do

Research Projects The mentors rated themselves lower than the students

Graph 2 compares how students and mentors feel about the level of proficiency that they have in certain areas as a result of the RISE Program Over all, students and mentors agree on the level of proficiency One additional area is Poster Presentation The two areas that needed work were layout and content The mean score was 7.14 out of a

possible 10 points for layout and 14 out of a possible 16 for content

Graph 3 compares the students and mentor's perceived awareness in several areas

following participation in the RISE Program

Awareness (p<.05) are of the Current Issues and Current Scholars in the Field The two

areas in which the differences between means are significant are in the mentors rated both

of these areas lower than the students

Finally, the students evaluated the summer program There were only three responses to the evaluation This makes analysis difficult Overall, however, students expressed that their summer experience helped them to gain an exposure to research and research

practices, develop critical thinking skills, and gave them practical experience in research skills All of the students said that they would recommend the program to another studentand would attend another research program in the sciences

Graph 3: Student/Mentor Comparason: Level of

The two areas in which the differences between means are significant (p<.05) are in the awareness are of the Current Issues and Current scholars in the field The mentors rated

both of these areas lower than the students

Finally, the students evaluated the summer program There were only three responses to the evaluation This makes analysis difficult Overall, however, students expressed that their summer experience helped them to gain an exposure to research and research practices, develop critical thinking skills, and gave them practical experience in research skills All of the students said that they would recommend the program to another student and would attend another research program in the sciences

RISE Evaluation : August 2001- January 2002

The following is an evaluation of RISE program activities for the 2001-2002 fall semester

Poster Presentation September 22, 2001

Seven MBRS-RISE students made poster presentation at the First Annual Fall Research Symposium held September 22, 2001 based on summer research experience The evaluator looked at the quality of the poster presentation and the confidence level of the presenters, using the same criteria developed for July 30th presentation In addition the quality of the poster was evaluated A Poster Presentation Evaluation Form was used to determine the quality of poster presentation Posters were judged on:

Four of the seven students who were not present at the July 30th presentation were observed for levels of confidence using the same criteria Confidence levels remained positive with all students meeting the five criteria.( minimum score 80)

Evaluation of Seminars, Workshops and Journal Clubs

The purpose of presentations, workshops, and seminars is to increase awareness of scientific fields of studies, expand knowledge base, provide role models and support research activities During the Fall Semester there were nine

presentations/workshops/seminars provided for RISE participants under the umbrella of the Emerging Caribbean Scientist (ECS) program

RISE students were asked to complete an evaluation of the seminars, workshops and journal clubs presented during the fall semester They were asked to respond to a questionnaire and (1) check the presentations they attended and rank them from 1 (low)

to 10 (high) (2) list one idea, concept, or information they learned and (3) their

opinion of the presenter The purpose of the evaluation form was:

to assess the respondent perception of the quality of the presentation , increase in

knowledge, the importance of presentation to their course of study, and respondent’s perception of presenters as experts, role models

Results

1 Attendance and Rankings

The three sessions that had the highest attendance and scored value were (1) the StudentResearch Symposium (9/22) (2) Approaches to HIV Vaccine Development (9/11) and (3) Workshop: Effective Poster Presentations (9/13) Ranked fourth were Workshop:

How to write an Abstract; (August 30) Seminar on Mathematical Models (August 28) ;

and Applying to Graduate School (October 25) Potluck for Faculty and Students (August31) ranked fifth, and Journal Club (October 19) and Special Workshop: Getting the Most Out of Professional Meetings (October 26) were last

Table 2 RISE STUDENTS’ RANKING OF SEMINARS/WORSHOPS FALL 2001

2 Approaches to HIV Vaccine Development September 11

3 Workshop: Effective Poster Presentations September 13

4 Workshop: How to write and Abstract

Seminar on Mathematical ModelsApplying to Graduate School

August 30August 28October 25

6 Journal Club on HIV

Special Workshop: Getting the Most Out of Professional Meetings

September 6 October 26

2 Ideas / Concepts/Thoughts and Perception of the Presenter

Participants were asked to describe/list one important idea, concept or information learned from the presentation There were eleven workshops/seminars presented on campus during the fall term Presentations were primarily subject or skill oriented

All respondents reported at least one concept/idea/information that they learned from their participation in the activity All presenters were rated positively Particular traits noted for faculty/visiting faculty led presentations were “easy to follow”, “explained clearly”, ‘well prepared, “very knowledgeable”, “understandable”, “presented quality research”, “work was timely” “informative.” Traits noted in student led presentations:

“well prepared”, “high level of student participation,” "presenters worked hard results convincing” “exhibited good traits we should emulate”

Criticism centered around lack of refreshments at students presentation (1), more

information needed on MCAT and additional information on professional schools (1) and not enough promotion of faculty/student potluck dinner (see appendix 3 ,Comments fromthe RISE Fellows on Program Evaluation Form)

ABSCMB Annual Conference and Metropolitan University Conference

Ten RISE Fellows attended the Annual Biomedical Research Conference for Minority Students (ABRCMS) (October 31-November 3) in Orlando Florida Additionally, Three RISE Fellows attended the Conference sponsored by Metropolitan University in San Juan Puerto Rico and supported by NFS funds The students made poster presentations at both Symposiums However, the ABRCMS conference was supported by the RISE grantand data were collected from RISE participants No data were collected by RISE on the UMet conference

The purpose of the ABRCMS conference is to expose minority student to careers in Biomedical Research , current topics in the field, and skills needed to be successful An eleven-part questionnaire was administered to participants by the RISE program to assess the value of participation in the conference Nine of the questions were to assess which sessions were most informative/appealing/beneficial and two related to travel/ planning issues.( appendix 4)

Ninety percent of the questionnaires were completed and returned Responses were analyzed to determine if the experience was perceived by participants as assisting in career choice/development and research ability

Results

All RISE students attending the conference attended sessions on career pathways Interest was shown in the session dealing with diseases that affect minorities with approximately 55 percent of the group attending RISE fellows perception of benefits from the symposium experience fell into four categories: career pathways/ development,,personal growth/ development, research opportunities, and networking Individual interests were also evident None of the RISE students visited the same poster

presentation and when asked which symposium was most informative/useful and why, notwo responses were exactly the same

Students were asked to rate( 1 [low] to 5 [high]) the degree to which their attendance at the symposium enhanced their knowledge of their field and ability to conduct research RISE fellows rated enhanced knowledge at 3.75 and research ability at 4.0 All RISE students responding agreed that the session on ethics was beneficial Responses indicatedthat students learned more about ethical conduct in research Example of responses : “I

learned that some issues which I had taken for granted are unethical”, “ it enabled me to understand that actions have consequences”, “ It let me know a whole lot more clearly what behavior is acceptable and what is not”

Responses to efficacy of planning varied The two questions (10, 11) related to what theyshould continue doing and what should be improved Students supported pre-meetings, which set expectations, planning related to tickets and hotel , and including

students/faculty Out of nine respondents, three left the question blank, one stated

“nothing I can think of,” and five gave suggestions Suggestions centered on issues of group cohesion and identity while traveling, ground transportation, and minimizing lost class time

Focus Groups and Interviews

Focus Groups

Two focus groups were held with Rise Fellows RISE students, including those who began in the RISE program during the summer but had moved on to be supported by other grants (MARC, NSF), and the ten students presently designated RISE Fellows and enrolled on both campuses.( ST Thomas and St Croix) The purpose of the focus group meetings was:

 To obtain information on the impact of the MBRS-RISE program on RISEFellows,

 To inform data previously collected through surveys

 To access the degree to which RISE objectives were being met

 Participants perception of the RISE program

 Problems, concerns that may impact success

 Suggestions for changes in implementation, objectives, for program improvements

The following questions framed the discussion:

Are there challenges to being a RISE Fellow?

 Did you have a research experience this summer and are you presently engaged inresearch? Describe it

 Did your research experience have an impact, influence your decision about pursuing work , continuing , going on in your field?

 Presentations/Seminars/National Conference How valuable were they? Are they worthwhile?

 Your professors and mentors How do you view them, professionally, support wise

 When faced with difficulty, or problems, possible failure how do you handle it? What motivates you to continue?

 Is being a RISE Fellow more work?

Two focus groups were held each for approximately one hour and ten minutes The first group included three RISE students from St Croix who participated via teleconference and four students on the St Thomas Campus The second group contained five students

on the St Thomas campus only RISE Fellows are comprised of students from physical sciences and social sciences

Ten of the thirteen students were involved in a research project Three students were new

to the program and have not been assigned a mentor or are working on a research project.These three students are presently tutors for students in gatekeeping courses

Analysis/Results of Focus Groups

1 Question: Is Being a RISE Fellow More Work? More Challenging?

Participants in both groups agreed that being a RISE Fellow was more work However, they believed it was worth it

1 “more advantages than disadvantages

2 “ More work/ Yes, [I] came back two weeks earlier, but it sets standards

3 “[it] motivates you

2 Question: What do you think of the Workshops/seminars? How valuable are they

to you.

All of the students supported the workshops and seminars However, there were areas of

concern Participants in the social sciences that not all were relevant although they saw a

connection between major (psychology) and biology One participant had a scheduling problem although, seminars are held from 12:00-1:00 Another was concerned that a

heavy credit load would interfere with his attendance

1 “ I’m a Psychology Major so all are not as relevant but some (those dealing with biology) gave me a more well rounded picture that I could apply to my work

2 Sometimes I felt left out (psychology)

3 “I don’t think they think of us (psychology) as science just the physical sciences

4 “they help us think more about graduate school

5 “Logistics I carry 18 credits with labs Sometimes hard but they are good”

6 “ I like interaction with professors” “ the ability to ask questions.”

7 “Sometimes hard to get to It’s at lunch time and I have classes scheduled

(Computer) I’m all the way over in CA.”

3 Question: When you face these challenges and things get tough, what motivates you? Do you ever feel like giving up?

Although students express that there are challenges, no discussion ensued on the difficulty they faced or the possibility of giving up Responses reflected inner

direction and self-actualization goals

1 “I have done a lot.[gives me] wider experience”

2 “[my] personal goals My career [aspirations]”

3 “Love for my field”

4 “feeling successful Being able to accomplish”

5 “Yes [but I think of }contributing to the whole community.”

4 Question: What about the Presentations here and in Orlando? Did you participate? What did you get from the experience?

This question garnered the most responses and enthusiasm Students spoke at length

on what they got from their participation, particularly the ABRCMS conference Discussions focus on seeing different career opportunities and opportunities for graduate schools as well as skills learned and knowledge enhanced

1 “great experience”

2 “I got to talk with schools like Harvard and Yale I would never had considered them”

3 “Learned a lot about presentations”

4 “ Improved my ability to speak to people about my work”

5 “Enriching –interacting with presenters”

6 “Enjoyable”

7 “[I have ] much more to learn”

8 “[ I liked] interaction with other students [from different places]

5 Question: What have you learned about research by being a RISE Fellow?

Students in both groups used opportunity to describe their research project and what they did They spoke about what they had learned both skills and knowledge Only one student had not done any research

1 “skills that are learned in research “[I learned to ] transcribe Stick to words subjects say”

2 “team work Group working together

3 “opened my mind to different things”

4 “Research is important-[it] changes society”

5 “no research [done] yet”

6 Career “Didn’t know what I wanted Now I do”

6 Question: How do you view your professors/mentors? Do you view them as experts in their field? Are they approachable?

Participants in both groups spoke positively about their mentors Both questions were responded to affirmatively There were no negative responses