Daten einlesen

Neue Daten können in der Instant transnormsilcm.ddnss.de erstellt werden. Hierfür mess lediglich ein Topic Model erstellt werden. Dabei wird dann automatisch die entsprechende Datei im Hauptorder gespiechert.

load("data_TM_Pruning_2020-03-16.RData")

Vokabulargrößen vor und nach dem Pruning

length(vocab_no_pruning)
## [1] 22414
length(vocab_pruning)
## [1] 9774

Anteil fehlender Worte pro Dokument durch Pruning

token_pro_dokument_vor_pruning<-Matrix::rowSums(dtm_glob)
token_pro_dokument_nach_pruning<-Matrix::rowSums(dtm)

percentage<-token_pro_dokument_nach_pruning/token_pro_dokument_vor_pruning
mean(percentage)
## [1] 0.9577198

Beispiel

# for doc id #1
dokument <- db_data$token[which(db_data$token$doc_id==rownames(dtm)[1]),] 
# assume tolower is active
words <- tolower(dokument$token)
active <- which(words%in%vocab_pruning)
lost_to_pruning <- which(words%in%(setdiff(vocab_no_pruning,vocab_pruning)))
lost_before_pruning <- which(!words%in% vocab_no_pruning)    

words[active]<-paste0("<span style='color:green';>",words[active],"</span>")
words[lost_to_pruning]<-paste0("<span style='color:blue';>",words[lost_to_pruning],"</span>")
words[lost_before_pruning]<-paste0("<span style='color:red';>",words[lost_before_pruning],"</span>")
text <- paste(words,collapse=" ")
#grün = vorhanden
#blau = beim Pruning entfernt
#rot = in vorherigen Schritten entfernt
shiny::HTML(text)
islamic_republic_of_afghanistan intended_nationally_determined_contribution submission to the united_nations framework convention on climate_change 21 september 2015 * * * * * the islamic_republic_of_afghanistan hereby_communicates its intended_nationally_determined_contribution ( indc ) and information to facilitate understanding of the contribution . executive summary base_year : 2005 target years : 2020 to 2030 contribution type : conditional sectors : energy , natural resource_management , agriculture , waste management and mining gases covered : carbon_dioxide ( co2 ) , methane ( ch4 ) , and nitrous_oxide ( n2o ) target : there will be a 13.6 % reduction in ghg_emissions by 2030 compared to a business as usual ( bau ) 2030 scenario , conditional on external support . financial needs : total : usd 17.405 billion adaptation : usd 10.785 billion mitigation : usd 6.62 billion ( 2020 - 2030 ) figure 1 figure 1 . greenhouse_gas emissions for afghanistan showing 13.6 % relative reduction in emissions compared to a business as usual scenario for the year 2030 1 1 . afghanistan ’s national_circumstances and commitment to climate_change introduction afghanistan has extensive development and climate_adaptation needs and , currently , low levels of greenhouse_gas ( ghg ) emissions . afghanistan remains one of the poorest countries in the world today , with an estimated population of 28.6 million ( 2015)1 and a per_capita gdp of usd 660.2 afghanistan is highly prone to natural_disasters throughout its 34 provinces.3 as a result of climate_change , it is anticipated that the incidence of extreme_weather events , including heat_waves , floods , and droughts will likely increase , as will climate_changelinked disasters such as glacial lake outflows . the majority of afghanistan ’s population relies directly or indirectly on the available natural_resources for their livelihoods so with these climatic changes the foundation of the country ’s economy , stability , and food_security is under threat . despite these challenges , afghanistan can remain a low emission economy while developing rapidly if , under the paris climate_change agreement , extensive financial and other resources are made available to allow afghanistan to successfully develop and implement low emission development strategies ( leds ) and highly effective adaptation and development strategies ( heads ) . appropriate support in the form of finance , capacity_building , technology and legal assistance is needed for afghanistan to make substantial progress on social and economic fronts while maintaining low per_capita ghg_emission levels . description of fairness and ambition afghanistan recognizes that all countries in the world need to make rapid progress towards lowering , or maintaining if already low , per_capita emission levels to avoid dangerous levels of global_warming . it is therefore critical that under the paris agreement , financial resources , capacity_building , technology_transfer , and other support is provided to afghanistan in order to enable it to implement leds and heads . this will allow afghanistan to continue developing while maintaining low level of emissions and increasing adaptive capacity to climate_change . there would be lower costs and a clearer development path for afghanistan if it pursued development using mainly fossil_fuels , as other countries have . this would likely result in ghg_emission in afghanistan continuing to increase at current rates for the period to 2025 and beyond . however , given the extremely limited remaining global ghg_emissions budget , afghanistan requires the unfccc , the global environmental facility ( gef ) , the green climate fund ( gcf ) , and other international institutional_arrangements to provide the extra finance and other support needed to successfully implement leds across all sectors of its economy without compromising socioeconomic_development_goals . national_development objectives despite suffering decades of instability and war , afghanistan has made considerable development progress . over the past 13 years notable achievements have been made in the areas of environment , agriculture , health , education , infrastructure , the economy , and the provision of other important basic services.4 afghanistan ’s national_development_strategy ( ands ) is based on afghanistan ’s millennium development_goals ( mdgs ) . it identifies the environment as “ a crosscutting issue that underpins the entire social and economic_development framework for the country . ” the national environment protection agency ( nepa ) , mandated to address environmental concerns , continues to work with all parts of the government of the islamic_republic_of_afghanistan ( giroa ) to mainstream environmental and climate 1 cso 2015 estimation 2 imf 2015 estimate for afghanistan . 3 nepa ( 2009 ) , national capacity needs self - assessment for global environmental management ( ncsa ) and national_adaptation programme of action for climate_change ( napa ) . 4 ( giroa , 2015 ) 2 change considerations into the country ’s national_development framework . nepa advocates for appropriate policies and measures that enable rapid development while ensuring good environmental and climate_change outcomes . afghanistan has developed its indc with the conviction that countering the effects of climate_change requires a commitment from all countries with regard to mitigation , and adaptation . in terms of international commitments on climate_change , afghanistan joined the unfccc in 1992 , and ratified the kyoto_protocol in 2013 . afghanistan completed its national_adaptation programmes of action for climate_change ( napa ) and national capacity needs selfassessment for global environmental management ( ncsa ) in 2009 . afghanistan submitted its initial national_communication ( inc ) under the unfccc in 2012 , and is currently preparing its second national_communication ( snc ) for submission to the unfccc in 2016 . at present , afghanistan is finalizing its national_climate_change strategy and action plan ( accsap ) as well as its national_adaptation_plan ( nap ) . in 2015 , as part of the indc preparation process , nepa convened a series of consultation and awarenessraising workshops to bring together decisionmakers from government institutions and stakeholders from nongovernmental organizations to develop the current indc and establish a sustainable_development vision for afghanistan . through this indc process and existing policies and strategies , the overall vision that was developed for afghanistan aims to enhance the adaptive capacity and resilience of its agriculture , environment , and population to climate_change , while developing and implementing leds . climate_change trends , impacts , and vulnerabilities afghanistan is ranked among the most vulnerable countries in the world to the adverse_impacts of climate_change.5 afghanistan ’s inc report has documented an increase of 0.6 ° c in the country ’s mean_annual temperature since 1960 . based upon recent climate observations , precipitation patterns have decreased during springtime ( march - may ) by approximately 40.5 mm ; however , the total annual precipitation has only slightly decreased by approximately 30 mm since there is a slight increase in precipitation from june until november . this implies that afghanistan is already beginning to experience the initial adverse_impacts of climate_change . recent climate projections , based on cordex regional climate models and representative concentration pathways ( rcps)6 , indicate that afghanistan will face an overall strong increase in mean_annual temperature , considerably higher than global mean projections , when compared to a baseline period of 1986 - 2006 . more specifically , under the “ optimistic ” scenario ( rcp4.5 ) , the mean of the model ensembles projects a warming of approximately 1.5 ° c until 2050 and of approximately 2.5 ° c until 2100 . for the “ pessimistic ” scenario ( rcp8.5 ) , the models project an extreme warming of approximately 3 ° c until 2050 , with further warming up to 7 ° c by 2100 . under both scenarios there are regional differences , with a higher temperature_increases at higher altitudes compared to the lowlands . for precipitation a significant ( α=0.05 ) mean decrease of precipitation during springtime ( march - may ) for the north , the central highlands and the east for both scenarios from 2006 until 2050 between 5 - 10 percent is seen . this decrease is offset by a slight increase of precipitation during autumn and wintertime ( october - december ) in these regions . for the hindu kush area , a significant and substantial increase in precipitation during the winter season of approximately 10 percent is seen , whereas during spring season precipitation is projected to stay stable . for the arid south of the country , the models do not project significant trends for precipitation . the decrease of precipitation during spring is particularly relevant since during these months the main plant growth for agricultural_production takes place . in addition , the decrease 5 dara climate_vulnerability monitor ( 2012 ) ; germanwatch global climate_risk index ( 2013 ) ; and notre dame global adaptation index ( 2014 ) . 6 representative concentration pathways ( rcps ) are ghg_emission scenarios adopted by the ipcc to describe four possible climate futures depending on the levels of future global ghgs emitted . there are four rcps : 1 ) rcp2.6 , which assumes that ghg_emissions peak between 2010 - 2020 and then decline ; 2 ) rcp4.5 , which assumes that ghg_emissions peak around 2040 and then decline ; 3 ) rcp6 , which assumes that ghg_emissions peak around 2080 and then decline ; and 4 ) rcp8.5 , which assumes that ghgs emissions continue to rise throughout the 21st century . 3 is projected to take place in the regions with the highest agricultural productivity of afghanistan ( east , north , and central highlands ) . in combination with the overall increase in temperature and the related increase in evapotranspiration across the country , this will most likely negatively impact the hydrological cycle , agricultural productivity , and availability of water resources . climatic changes are also likely to impact upon the spread of water , food and vectorborne diseases , presenting considerable health risks to both urban and rural populations . finally , the aforementioned climateinduced risks and challenges can enhance social inequalities , poverty , and food_insecurity causing considerable and fundamental threats to human life , livelihoods , property , political stability , the economy , and the environment in afghanistan . based upon these climate_change projections , afghanistan ’s environment will experience considerable changes over the remainder of this century . climate_change , based on sound scientific analysis of climatic changes and uncertainties , must be mainstreamed into sectoral planning to reduce the negative_impacts of climate_change in afghanistan and increase resilience , both in rural and urban areas . 2 . climate_change adaptation near- and longterm adaptation visions , goals and targets afghanistan ’s vision for addressing the adverse_impacts of climate_change through adaptation aims to protect the country and its population by enhancing adaptive capacity and resilience , effectively respond to the vulnerabilities of critical sectors , and efficiently mainstream climate_change considerations into national_development policies , strategies , and plans . in order to achieve this vision , a national_strategy for climate_change adaptation must include community level vulnerabilities and build up their adaptive capacities by investment in short- and longterm initiatives . shortterm action_plans formed part of the 2009 napa , while the nap will implement both short- and longterm priorities these priorities include , but are not limited to : 1 . reducing vulnerability of the country and its population through enhancement of adaptive capacity and resilience , and deployment of disaster risk reduction approaches 2 . integrating climate_change consideration into the national_planning_processes 3 . promoting economic_development and sustainable rural livelihoods through sustainable_management of environmental resources and increase access to modern forms of efficient and sustainable energy services 4 . improvement of technical capacity in governmental institutions 5 . adaptive and integrated land and water_management 6 . improving access by rural communities and farmers to water to support food_security , reduce poverty and improve agricultural_productions 7 . raising awareness for people of afghanistan on climate_change impacts and adaptation measures current adaptation undertakings and support afghanistan has initiated a number of steps to promote the country ’s sustainable_development . in term of national_development policies , plans , and legal_frameworks , considerable effort has been placed on addressing environmental challenges , disaster risk reduction , food_security , water_security , protection of forest and rangelands , and biodiversity_conservation , all of which have clear relevance to climate_change adaptation . some noteworthy examples of such policies and plans that have successfully and explicitly integrated climate_change include : afghanistan national renewable energy policy ( anrep ) national water and natural resource_management priority programme strategic national_action plan for disaster risk reduction ( snap ) national environmental action plan ( neap ) national comprehensive agriculture production and market development programme energy for rural development ( erda ) national biodiversity strategy and action plan ( nbsap ) 4 additional national_development policies , strategies , and plans that currently do not mention climate_change but have entry points for the further mainstreaming of climate_change include : national agricultural development framework ( nadf ) national environment strategy energy sector strategy national forestry management plan rangeland management plan strategic policy framework for the water sector in term of programmes and projects , support provided by the international_community and multilateral agencies have laid the groundwork for building afghanistan ’s adaptive capacity and resilience to climate_change . major contributors include the tokyo framework bilateral partners , global environmental facility ( gef ) , which has provided support through enabling activities , midsize projects , and fullsize climate_change adaptation_projects funded by the least_developed_countries fund ( ldcf ) , as well as bilateral donors , nongovernmental organizations , and the united_nations . with the scale and urgency of afghanistan ’s adaptation needs , additional financial and other resources are strongly needed in order to effectively build the adaptive capacity and resilience of the country and its people before more severe impacts of climate_change begin to be felt . adaptation needs and means of implementation ( moi ) afghanistan faces a number of specific challenges in terms of addressing climate_change through adaptation . these challenges include , but are not limited to , funding gaps , lack of expertise , lack of reliable historical climate data , weak public_awareness about environmental issues , and security . afghanistan has identified the following key actions as part of its national_adaptation_plan ( nap ) in order to overcome existing gaps and barriers towards sufficiently addressing its climate_change adaptation needs . the total estimated cost of full implementation of the nap is usd10.785 billion over ten years . action- planning , technology capacity_building finance needs technology needs and capacity_building needs needs ( usd ) development and adoption of the own contribution afghanistan ccsap . -- -- development of a system to climate science climate science institutes 0.02 billion monitor and assess vulnerability technology with university and adaptation to climate_change . identification and mainstreaming climate_policy training afghan climate 0.01 billion of climate_change adaptation technologies and policy experts technologies into the sectoral methods policies , strategies and development_plans , and promotion of regional and international_cooperation and coordination for adaptation technology_transfer . strengthen and expand hydrological , operators and analysts for 0.1 billion meteorological and hydrological meteorological and hydrological , monitoring networks and data equipment and meteorological and data services , including a national integrated systems integrated systems database to archive and store meteorological and hydrological data . development of water resources improved designs and ecological engineering and 0.75 billion through rehabilitation and methodologies for spatial planning for water reconstruction of small- , catchment resources medium- , and largescale management 5 infrastructure . technology planning for proper watershed full catchment practitioners for watershed 2.5 billion management and promoted planning technology management through communitybased natural and models resources_management . increasing irrigated agricultural ecoagriculture and vocational and 4.5 billion land to 3.14 mha , through climate friendly engineering capacity to restoration and development of irrigation technology design , build and maintain afghanistan ’s irrigation systems . transfer to climate friendly irrigation afghanistan networks and local schemes . at least 10 % of afghanistan land conservation ecology protected_areas and species 0.3 billion area and the habitat of selected methods and tools ecologists , and ecological species under a system of economists trained and conservation working . behavioural change and technology_transfer national centre for 0.105 billion opportunities for provision and of renewable energy sustainable energy development of alternative and and sustainable strengthened and renewable energy_sources for energy expanded . combine public 25 % of the rural population and private competencies . above existing levels ( 15 % ) , in order to contribute to a reduction in the unsustainable usage of natural_resources and decreasing the strong reliance on fossil_fuels by rural communities . regeneration of at least 40 % of forestry and practitioners group built in 2.5 billion existing degraded forests and rangeland university , government rangeland_areas ( the area covered management tools and and local delivery levels . will be approximately 232,050 ha methods transferred to for forestry ; and 5.35 million ha afghanistan for rangelands ) . total financial resources needed : 10.785 billion 3 . climate_change mitigation ghg_emissions and mitigation measures afghanistan has very low relative per_capita ghg_emissions . while 1990 emissions were at 0.2 metric tons co2 per_capita , data indicates that per_capita emissions were around 0.3 for 2010 , making afghanistan one of the lowest ghg emitters globally.7 however , the country is on a growth path , which is expected to strengthen over the coming years , meaning ghg_emissions are likely to increase . it is important that support be provided to afghanistan to develop leds to minimize the increase in its ghg_emissions . afghanistan ’s overall ghg_emission figures ( table 1 ) demonstrate that the most important sources of co2 emissions are from the “ land - use change and forestry ” and “ energy ” sectors.8 in terms of ch4 and n2o emissions , the agriculture_sector is the major contributor.9 it is therefore important that greater attention should be focused on ghg mitigation measures within these sectors . afghanistan has only started to access unfccc technology_transfer opportunities . it needs to build national capacity to navigate opportunities and play a more critical role in international negotiations . the following mitigation_options are designed to enable afghanistan to make a mitigation contribution which is condition on support needs for financial and technical_support being met . 7 the world_bank , co2 emissions ( metric tons per_capita ) . http://data.worldbank.org/indicator/en.atm.co2e.pc 8 nepa & adb ( 2007 ) , afghanistan greenhouse_gas inventory report . 9 ibid 6 2030 mitigation contribution type of conditional target : a relatively reduction in ghg_emission is achievable through contribution meeting afghanistan ’s financial , technical , and technological needs in energy , forest and rangeland , industrial process and extractive industry , agriculture and livestock , and waste management sectors . primarily focused is on sustainable process and development initiatives based on the outcomes of 2015 national consultation on leds and nama . gases carbon_dioxide ( co2 ) , methane ( ch4 ) , and nitrous_oxide ( n2o ) sectors energy production ( hydropower , solar systems , wind and biomass , commercial , domestic : clean cook_stoves and fuels , and solar energy energy_efficiency ( households , transport , industry , services , mining , agriculture ) land_use , forests and rangelands ( afforestation and reforestation , natural forests , fuelwood from forest and orchards , rangelands rehabilitation ) agriculture and livestock ( manure management , land_use / change for agriculture ) irrigation infrastructure improved cropping systems industrial processes and extractive industries ( mining and extractives , gas and hydrocarbons , coal and minerals ) enduse saving , and fuel shifts power_plants : fuel shift to natural_gas and renewables transport : more efficient vehicles , clean fuels , and alternative fuels waste management ( solid waste management and wastewater recycling / composing of biodegradable waste instead of landfill , and methane recovery from landfill ) coal mines : gas recovery in coal mines rice paddies : modified rice strains mitigation measures for n2o include reduced fertilizer application ; optimal timing of fertilizer application ; nitrification inhibitors , less use of histosols ( peat soils)10 . afghanistan ’s ghg inventory in table 1 is calculated for each ghg_emission sector for 200511 and businessas - usual projections based on the growth_rate for a mediumterm timeframe ( 2020 - 2030 ) . table 1 : green house gas emissions of co2 , ch4 and n20 in afghanistan in 2005 - 2030 * ghg_emission co2 equivalent , gg 2020 2025 2030 sector co2 ch4 n2o aggregated co2-eq , gg co2-eq , gg co2-eq , gg energy 2,910.04 736.00 129.83 3,775.87 9,745.46 10,849.02 12,087.00 industry 312.15 - - 312.15 791.57 878.25 974.42 agriculture - 9,296.49 5,812.50 15,108.99 24,665.30 * * 29,578.77 * * 35,471.04 * * land_use change and 9,341.13 80.64 9.30 9,431.07 10,949.18 11,507.70 12,094.71 forestry waste - 130.41 - 130.41 330.70 * * 366.91 * * 407.09 * * total ghg_emission 12,563.32 10,243.54 5,951.63 28,758.49 46,482.20 53,180.64 61,034.25 incl . lulucf total ghg_emission 3,222.19 10,162.90 5,942.33 19,327.42 35,533.02 41,672.95 48,939.54 excl . lulucf * information used from adb – afghanistan greenhouse_gas inventory report and projection for 2020 - 2030 using gacmo model * * ch4 ( ch4 emission x 21 ) and n2o ( n2o emission x 310 ) counted as co2-eq 10 ibid 11 ibid 7 climate mitigation gaps and barriers and support needs usd 662 million / year from 2020 finance sector technology and capacity_building needs needs ( usd ) energy_efficiency in carbon finance and project development skills . 100 buildings and in information on available technologies , measures , and financing skills . million / year transport_sector traditional customs and administered pricing . building codes , and standards on appliances and equipment . clean cooking , heating and power projects . energy human and institutional capacity for adoption of cleaner technology . 188 capital markets that encourage investment in decentralized systems . million / year information and intellectual property rights for mitigation technologies . renewable energy , entry costs support , access to capital , and subsidies . environmental compliance standards ( emission and indoor ) . waste management landfill management , decentralised wastewater treatment . 74 climate project development skills . million / year forest and carbon_sequestration on forest / rangelands , and forest carbon skills . 100 rangelands funding institutional capacity to monitor and verify projects . million / year better spatial planning for community and production agriculture . reduce rural peoples’ dependence on fuel for cooking and heating . industry and mining cleaner coal mining , leaveit - inthe - ground approaches , combustion , and 100 transportation of minerals . million / year hydrocarbon fields management . technical industrial capacity to link basic industry and mining private and public sector with climate sector experts . agriculture and national herd , reduction in fuel used , or cleaner fuel technologies . 100 livestock southsouth collaboration on lowcarbon agriculture , study tours . million / year funding for r&d activities . improved national dataset on agriculture , food_security data . 8

Insgesamter Anteil durch Pre-Processing entfernter Wortformen (types)

# types
# Vokabulargröße ohne Preprocessing
# ohne tolower:
length(unique(db_data$token$token))
## [1] 36194
# mit tolower:
length(unique(tolower(db_data$token$token)))
## [1] 30915
# Untersuchung pro Dokument
doc_ids<-rownames(dtm)
types_per_doc_before_preprocessing <- unlist(lapply(X = doc_ids,FUN = function(x){
  length(unique(db_data$token$token[which(db_data$token$doc_id==x)]))
})
)

types_per_doc_after_preprocessing <- unlist(lapply(1:nrow(dtm),FUN = function(x){
  length(which(dtm[x,]>0))
})
)

# durchschnittlicher Anteil
mean(types_per_doc_after_preprocessing/types_per_doc_before_preprocessing)
## [1] 0.7118066
hist(x = types_per_doc_after_preprocessing/types_per_doc_before_preprocessing,breaks = 20,border = "red",
     main = "histogram of resulting precentages of types per document")

Insgesamter Anteil durch Pre-Processing entfernter Worte (token)

# token
# Anzahl token ohne Preprocessing
length(db_data$token$token)
## [1] 1839543
# Anzahl token nach preprocessing
sum(dtm)
## [1] 765921
# Untersuchung pro Dokument
doc_ids<-rownames(dtm)
token_per_doc_before_preprocessing <- unlist(lapply(X = doc_ids,FUN = function(x){
  length(db_data$token$token[which(db_data$token$doc_id==x)])
})
)

token_per_doc_after_preprocessing <- unlist(lapply(1:nrow(dtm),FUN = function(x){
  sum(dtm[x,])
})
)

# durchschnittlicher Anteil
mean(token_per_doc_after_preprocessing/token_per_doc_before_preprocessing)
## [1] 0.4194171
hist(x = token_per_doc_after_preprocessing/token_per_doc_before_preprocessing,breaks = 20,border = "red",
     main = "histogram of resulting precentages of token per document")